[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:59:26.516255 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43861,127.0.99.61:38161,127.0.99.60:40345
I20240503 07:59:26.517352 396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:26.517907 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:26.526261 404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.526978 402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.527776 401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.528533 396 server_base.cc:1034] running on GCE node
I20240503 07:59:26.529109 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.529239 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.529279 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166529278 us; error 0 us; skew 500 ppm
I20240503 07:59:26.536540 396 webserver.cc:460] Webserver started at http://127.0.99.62:46417/ using document root <none> and password file <none>
I20240503 07:59:26.537139 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.537209 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.537441 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.539218 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "417af0034cab4f96ab54b49605bd884e"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.542827 396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:26.545076 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.546224 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:26.546340 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "417af0034cab4f96ab54b49605bd884e"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.546435 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.562381 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.563050 396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:26.563239 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.569986 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43861
I20240503 07:59:26.570003 461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43861 every 8 connection(s)
I20240503 07:59:26.571139 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:26.573608 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:26.573757 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.574607 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.575392 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.576297 396 server_base.cc:1034] running on GCE node
I20240503 07:59:26.576496 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.576536 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.576552 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166576552 us; error 0 us; skew 500 ppm
I20240503 07:59:26.577665 396 webserver.cc:460] Webserver started at http://127.0.99.61:36479/ using document root <none> and password file <none>
I20240503 07:59:26.577811 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.577855 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.577921 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.578250 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root/instance:
uuid: "8f7f2f0f7e414fbca47b19678bf04f00"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.578377 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.579939 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:26.581004 473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.581350 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.581413 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
uuid: "8f7f2f0f7e414fbca47b19678bf04f00"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.581485 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.588851 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
W20240503 07:59:26.589612 413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.61:38161: connect: Connection refused (error 111)
W20240503 07:59:26.590482 462 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:38161: Network error: Client connection negotiation failed: client connection to 127.0.99.61:38161: connect: Connection refused (error 111)
I20240503 07:59:26.594005 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.594370 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.599076 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:38161
I20240503 07:59:26.599131 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:38161 every 8 connection(s)
I20240503 07:59:26.599738 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:26.599918 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.601567 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
W20240503 07:59:26.602523 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.603366 396 server_base.cc:1034] running on GCE node
W20240503 07:59:26.604084 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.604703 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.604985 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.605046 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.605067 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166605067 us; error 0 us; skew 500 ppm
I20240503 07:59:26.605943 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.606096 396 webserver.cc:460] Webserver started at http://127.0.99.60:43615/ using document root <none> and password file <none>
I20240503 07:59:26.606258 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.606307 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.606380 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.606814 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root/instance:
uuid: "2f65fde336c04151b3b1fd8f23744e4d"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.608258 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:26.609360 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.609609 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.609730 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
uuid: "2f65fde336c04151b3b1fd8f23744e4d"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.609810 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.610230 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
W20240503 07:59:26.611436 528 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:40345: Network error: Client connection negotiation failed: client connection to 127.0.99.60:40345: connect: Connection refused (error 111)
I20240503 07:59:26.616658 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.617025 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.621781 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:40345
I20240503 07:59:26.621819 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:40345 every 8 connection(s)
I20240503 07:59:26.622187 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 07:59:26.622653 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.623688 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.627875 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.631342 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.637326 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: Bootstrap starting.
I20240503 07:59:26.639693 462 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } attempt: 1
I20240503 07:59:26.639698 594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.640779 594 log.cc:824] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:26.642397 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: No bootstrap required, opened a new log
I20240503 07:59:26.643651 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:26.646134 594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.646303 594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.646361 594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f65fde336c04151b3b1fd8f23744e4d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.646755 594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.647194 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.647310 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.647853 594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:26.649363 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: Bootstrap starting.
I20240503 07:59:26.650645 462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.651670 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: No bootstrap required, opened a new log
I20240503 07:59:26.652027 462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.652110 462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.652144 462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417af0034cab4f96ab54b49605bd884e, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.652299 462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.652582 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.652717 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: This master's current role is: FOLLOWER
W20240503 07:59:26.652805 610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 07:59:26.652793 462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: configured and running, proceeding with master startup.
W20240503 07:59:26.652926 610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:59:26.654542 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 07:59:26.655062 624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:26.655120 624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:59:26.663604 528 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } attempt: 1
I20240503 07:59:26.669131 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: Bootstrap starting.
I20240503 07:59:26.670205 528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.671726 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: No bootstrap required, opened a new log
I20240503 07:59:26.672240 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.672358 528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.672394 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7f2f0f7e414fbca47b19678bf04f00, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.672519 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.672827 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.672917 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.673004 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:26.674925 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 07:59:26.675568 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:26.675645 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:26.675697 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 07:59:26.678802 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.679615 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.680531 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.681291 396 server_base.cc:1034] running on GCE node
I20240503 07:59:26.681484 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.681535 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.681569 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166681568 us; error 0 us; skew 500 ppm
I20240503 07:59:26.682678 396 webserver.cc:460] Webserver started at http://127.0.99.1:38049/ using document root <none> and password file <none>
I20240503 07:59:26.682844 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.682893 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.682963 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.683401 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "8c96d1d521954c79bd4840dc14893a37"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.685140 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:26.686311 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.686610 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:26.686688 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "8c96d1d521954c79bd4840dc14893a37"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.686756 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.699721 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.700568 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.701004 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:26.701926 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:26.701989 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.702034 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:26.702073 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.708509 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:39307
I20240503 07:59:26.708546 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:39307 every 8 connection(s)
I20240503 07:59:26.709679 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:26.713241 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.713871 396 server_base.cc:1034] running on GCE node
W20240503 07:59:26.714618 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.715540 711 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43861
I20240503 07:59:26.715727 710 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40345
I20240503 07:59:26.715772 711 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.715795 710 heartbeater.cc:464] Registering TS with master...
W20240503 07:59:26.715518 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.716437 710 heartbeater.cc:511] Master 127.0.99.60:40345 requested a full tablet report, sending...
I20240503 07:59:26.716686 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.716980 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.717015 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166717013 us; error 0 us; skew 500 ppm
I20240503 07:59:26.717996 557 ts_manager.cc:194] Registered new tserver with Master: 8c96d1d521954c79bd4840dc14893a37 (127.0.99.1:39307)
I20240503 07:59:26.718381 396 webserver.cc:460] Webserver started at http://127.0.99.2:42883/ using document root <none> and password file <none>
I20240503 07:59:26.716437 711 heartbeater.cc:511] Master 127.0.99.62:43861 requested a full tablet report, sending...
I20240503 07:59:26.718533 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.718590 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.718603 712 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38161
I20240503 07:59:26.718652 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.718668 712 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.718860 712 heartbeater.cc:511] Master 127.0.99.61:38161 requested a full tablet report, sending...
I20240503 07:59:26.719092 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "9ec08b404c1f45f5bdb996289b593b78"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.719142 426 ts_manager.cc:194] Registered new tserver with Master: 8c96d1d521954c79bd4840dc14893a37 (127.0.99.1:39307)
I20240503 07:59:26.719552 492 ts_manager.cc:194] Registered new tserver with Master: 8c96d1d521954c79bd4840dc14893a37 (127.0.99.1:39307)
I20240503 07:59:26.720921 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:26.721962 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.722232 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.722313 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "9ec08b404c1f45f5bdb996289b593b78"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.722378 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.734318 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.734777 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.735117 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:26.735646 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:26.735698 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.735740 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:26.735766 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.742342 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:44577
I20240503 07:59:26.742414 788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:44577 every 8 connection(s)
I20240503 07:59:26.743352 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:26.747263 796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:26.747978 797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.749050 789 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40345
I20240503 07:59:26.749239 789 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.749501 790 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43861
I20240503 07:59:26.749543 396 server_base.cc:1034] running on GCE node
I20240503 07:59:26.749603 790 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.749904 790 heartbeater.cc:511] Master 127.0.99.62:43861 requested a full tablet report, sending...
W20240503 07:59:26.749933 799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:26.749512 789 heartbeater.cc:511] Master 127.0.99.60:40345 requested a full tablet report, sending...
I20240503 07:59:26.750357 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:26.750442 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:26.750525 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723166750524 us; error 0 us; skew 500 ppm
I20240503 07:59:26.750936 557 ts_manager.cc:194] Registered new tserver with Master: 9ec08b404c1f45f5bdb996289b593b78 (127.0.99.2:44577)
I20240503 07:59:26.751230 426 ts_manager.cc:194] Registered new tserver with Master: 9ec08b404c1f45f5bdb996289b593b78 (127.0.99.2:44577)
I20240503 07:59:26.751986 791 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38161
I20240503 07:59:26.752055 396 webserver.cc:460] Webserver started at http://127.0.99.3:37429/ using document root <none> and password file <none>
I20240503 07:59:26.752066 791 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.752259 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:26.752321 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:26.752372 791 heartbeater.cc:511] Master 127.0.99.61:38161 requested a full tablet report, sending...
I20240503 07:59:26.752398 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:26.752943 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "a48f9180ceed4835baadaae5bfd6f116"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.753044 492 ts_manager.cc:194] Registered new tserver with Master: 9ec08b404c1f45f5bdb996289b593b78 (127.0.99.2:44577)
I20240503 07:59:26.754747 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:26.755659 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.755903 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.755962 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "a48f9180ceed4835baadaae5bfd6f116"
format_stamp: "Formatted at 2024-05-03 07:59:26 on dist-test-slave-mxwh"
I20240503 07:59:26.756026 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:26.788372 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:26.788839 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:26.789186 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:26.789757 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:26.789810 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.789862 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:26.789896 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:26.796829 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:44793
I20240503 07:59:26.796876 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:44793 every 8 connection(s)
I20240503 07:59:26.799320 626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.799491 626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.800413 626 leader_election.cc:288] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 417af0034cab4f96ab54b49605bd884e (127.0.99.62:43861), 2f65fde336c04151b3b1fd8f23744e4d (127.0.99.60:40345)
I20240503 07:59:26.801288 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417af0034cab4f96ab54b49605bd884e" is_pre_election: true
I20240503 07:59:26.801491 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f7f2f0f7e414fbca47b19678bf04f00 in term 0.
I20240503 07:59:26.801810 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f65fde336c04151b3b1fd8f23744e4d" is_pre_election: true
I20240503 07:59:26.801924 568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f7f2f0f7e414fbca47b19678bf04f00 in term 0.
I20240503 07:59:26.802340 478 leader_election.cc:302] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f65fde336c04151b3b1fd8f23744e4d, 8f7f2f0f7e414fbca47b19678bf04f00; no voters:
I20240503 07:59:26.802986 626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.803053 626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.803094 626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.803311 868 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40345
I20240503 07:59:26.803398 868 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.803643 868 heartbeater.cc:511] Master 127.0.99.60:40345 requested a full tablet report, sending...
I20240503 07:59:26.804097 626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.804342 557 ts_manager.cc:194] Registered new tserver with Master: a48f9180ceed4835baadaae5bfd6f116 (127.0.99.3:44793)
I20240503 07:59:26.804433 870 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38161
I20240503 07:59:26.804462 626 leader_election.cc:288] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [CANDIDATE]: Term 1 election: Requested vote from peers 417af0034cab4f96ab54b49605bd884e (127.0.99.62:43861), 2f65fde336c04151b3b1fd8f23744e4d (127.0.99.60:40345)
I20240503 07:59:26.804507 870 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.804703 870 heartbeater.cc:511] Master 127.0.99.61:38161 requested a full tablet report, sending...
I20240503 07:59:26.804966 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f65fde336c04151b3b1fd8f23744e4d"
I20240503 07:59:26.805003 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417af0034cab4f96ab54b49605bd884e"
I20240503 07:59:26.805095 568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.805150 436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.805305 492 ts_manager.cc:194] Registered new tserver with Master: a48f9180ceed4835baadaae5bfd6f116 (127.0.99.3:44793)
I20240503 07:59:26.806066 568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f7f2f0f7e414fbca47b19678bf04f00 in term 1.
I20240503 07:59:26.806066 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f7f2f0f7e414fbca47b19678bf04f00 in term 1.
I20240503 07:59:26.806458 475 leader_election.cc:302] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417af0034cab4f96ab54b49605bd884e, 8f7f2f0f7e414fbca47b19678bf04f00; no voters:
I20240503 07:59:26.806712 626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.806936 626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 1 LEADER]: Becoming Leader. State: Replica: 8f7f2f0f7e414fbca47b19678bf04f00, State: Running, Role: LEADER
I20240503 07:59:26.807214 626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } }
I20240503 07:59:26.809379 869 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43861
I20240503 07:59:26.809476 869 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:26.809676 869 heartbeater.cc:511] Master 127.0.99.62:43861 requested a full tablet report, sending...
I20240503 07:59:26.809625 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7f2f0f7e414fbca47b19678bf04f00. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.809746 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:26.810317 426 ts_manager.cc:194] Registered new tserver with Master: a48f9180ceed4835baadaae5bfd6f116 (127.0.99.3:44793)
I20240503 07:59:26.810338 877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:26.810765 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013317229s
I20240503 07:59:26.811683 877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240503 07:59:26.814392 415 tablet.cc:2369] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:26.816676 436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 1 FOLLOWER]: Refusing update from remote peer 8f7f2f0f7e414fbca47b19678bf04f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.816676 568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 1 FOLLOWER]: Refusing update from remote peer 8f7f2f0f7e414fbca47b19678bf04f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.817266 626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.817739 626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.819130 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7f2f0f7e414fbca47b19678bf04f00. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.819139 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f7f2f0f7e414fbca47b19678bf04f00. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.819427 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.819412 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.820495 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.820616 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:26.820873 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.820959 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.821617 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.821712 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:26.822254 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417af0034cab4f96ab54b49605bd884e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 43861 } } peers { permanent_uuid: "8f7f2f0f7e414fbca47b19678bf04f00" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38161 } } peers { permanent_uuid: "2f65fde336c04151b3b1fd8f23744e4d" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40345 } } }
I20240503 07:59:26.822343 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:26.823442 877 catalog_manager.cc:1347] Generated new cluster ID: 7851e98c14f549bfa55595fc0050c2b1
I20240503 07:59:26.823500 877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:26.837780 877 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:26.838927 877 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:26.848482 877 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: Generated new TSK 0
I20240503 07:59:26.849419 877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:27.654080 610 catalog_manager.cc:1259] Loaded cluster ID: 7851e98c14f549bfa55595fc0050c2b1
I20240503 07:59:27.654170 610 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: loading cluster ID for follower catalog manager: success
I20240503 07:59:27.655441 610 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: acquiring CA information for follower catalog manager: success
I20240503 07:59:27.655880 624 catalog_manager.cc:1259] Loaded cluster ID: 7851e98c14f549bfa55595fc0050c2b1
I20240503 07:59:27.655937 624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: loading cluster ID for follower catalog manager: success
I20240503 07:59:27.656250 610 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 07:59:27.657178 624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: acquiring CA information for follower catalog manager: success
I20240503 07:59:27.657673 624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 07:59:27.721891 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51614
I20240503 07:59:27.754524 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51622
I20240503 07:59:27.807165 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51636
I20240503 07:59:28.675120 636 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:28.723110 712 heartbeater.cc:503] Master 127.0.99.61:38161 was elected leader, sending a full tablet report...
I20240503 07:59:28.755340 791 heartbeater.cc:503] Master 127.0.99.61:38161 was elected leader, sending a full tablet report...
I20240503 07:59:28.808082 870 heartbeater.cc:503] Master 127.0.99.61:38161 was elected leader, sending a full tablet report...
I20240503 07:59:28.965269 396 test_util.cc:268] Using random seed: 425710224
I20240503 07:59:28.978075 492 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51652:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 07:59:28.995651 753 tablet_service.cc:1467] Processing CreateTablet for tablet 104f4fd670784a1aa61fab763cadbaf9 (DEFAULT_TABLE table=test-workload [id=4e4b3cd2df524dcf842eb4ba47a2278f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 07:59:28.995710 674 tablet_service.cc:1467] Processing CreateTablet for tablet 13a1a265d0c14450b83661c705bb0443 (DEFAULT_TABLE table=test-workload [id=4e4b3cd2df524dcf842eb4ba47a2278f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 07:59:28.995738 673 tablet_service.cc:1467] Processing CreateTablet for tablet 64f0cee18c444e3da5965dc24c05f8c4 (DEFAULT_TABLE table=test-workload [id=4e4b3cd2df524dcf842eb4ba47a2278f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 07:59:28.996191 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a1a265d0c14450b83661c705bb0443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.996623 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f0cee18c444e3da5965dc24c05f8c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.996752 832 tablet_service.cc:1467] Processing CreateTablet for tablet fa59484a1caa40a6b1ce21caad45c2ee (DEFAULT_TABLE table=test-workload [id=4e4b3cd2df524dcf842eb4ba47a2278f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 07:59:28.996973 832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa59484a1caa40a6b1ce21caad45c2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.998205 917 tablet_bootstrap.cc:492] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: Bootstrap starting.
I20240503 07:59:28.996191 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104f4fd670784a1aa61fab763cadbaf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.999012 918 tablet_bootstrap.cc:492] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: Bootstrap starting.
I20240503 07:59:28.999377 917 tablet_bootstrap.cc:654] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.999934 918 tablet_bootstrap.cc:654] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.000481 917 tablet_bootstrap.cc:492] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: No bootstrap required, opened a new log
I20240503 07:59:29.000587 917 ts_tablet_manager.cc:1367] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.000943 918 tablet_bootstrap.cc:492] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: No bootstrap required, opened a new log
I20240503 07:59:29.000938 917 raft_consensus.cc:348] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.001034 917 raft_consensus.cc:374] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.001041 918 ts_tablet_manager.cc:1367] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.001098 917 raft_consensus.cc:729] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c96d1d521954c79bd4840dc14893a37, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.001236 917 consensus_queue.cc:260] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.001325 917 raft_consensus.cc:388] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.001377 917 raft_consensus.cc:482] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.001379 918 raft_consensus.cc:348] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f9180ceed4835baadaae5bfd6f116" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44793 } }
I20240503 07:59:29.001513 917 raft_consensus.cc:3037] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.001521 918 raft_consensus.cc:374] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.001521 920 tablet_bootstrap.cc:492] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: Bootstrap starting.
I20240503 07:59:29.001612 918 raft_consensus.cc:729] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48f9180ceed4835baadaae5bfd6f116, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.001750 918 consensus_queue.cc:260] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f9180ceed4835baadaae5bfd6f116" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44793 } }
I20240503 07:59:29.001829 918 raft_consensus.cc:388] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.001870 918 raft_consensus.cc:482] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.001926 918 raft_consensus.cc:3037] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.002364 917 raft_consensus.cc:504] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.002460 920 tablet_bootstrap.cc:654] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.002511 917 leader_election.cc:302] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c96d1d521954c79bd4840dc14893a37; no voters:
I20240503 07:59:29.002548 918 raft_consensus.cc:504] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f9180ceed4835baadaae5bfd6f116" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44793 } }
I20240503 07:59:29.002673 918 leader_election.cc:302] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48f9180ceed4835baadaae5bfd6f116; no voters:
I20240503 07:59:29.002720 917 leader_election.cc:288] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.002876 918 leader_election.cc:288] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.003080 917 ts_tablet_manager.cc:1398] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 07:59:29.003170 918 ts_tablet_manager.cc:1398] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.003451 920 tablet_bootstrap.cc:492] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: No bootstrap required, opened a new log
I20240503 07:59:29.003530 920 ts_tablet_manager.cc:1367] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.003746 922 raft_consensus.cc:2781] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.003850 920 raft_consensus.cc:348] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec08b404c1f45f5bdb996289b593b78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44577 } }
I20240503 07:59:29.003943 920 raft_consensus.cc:374] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.003978 920 raft_consensus.cc:729] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec08b404c1f45f5bdb996289b593b78, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.004096 922 raft_consensus.cc:686] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 1 LEADER]: Becoming Leader. State: Replica: 8c96d1d521954c79bd4840dc14893a37, State: Running, Role: LEADER
I20240503 07:59:29.004123 920 consensus_queue.cc:260] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec08b404c1f45f5bdb996289b593b78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44577 } }
I20240503 07:59:29.004289 920 raft_consensus.cc:388] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.004336 920 raft_consensus.cc:482] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.004377 920 raft_consensus.cc:3037] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.004487 922 consensus_queue.cc:237] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.004703 917 tablet_bootstrap.cc:492] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: Bootstrap starting.
I20240503 07:59:29.005239 920 raft_consensus.cc:504] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec08b404c1f45f5bdb996289b593b78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44577 } }
I20240503 07:59:29.005358 920 leader_election.cc:302] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec08b404c1f45f5bdb996289b593b78; no voters:
I20240503 07:59:29.005537 920 leader_election.cc:288] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.005666 929 raft_consensus.cc:2781] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.005831 920 ts_tablet_manager.cc:1398] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:29.005940 929 raft_consensus.cc:686] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec08b404c1f45f5bdb996289b593b78, State: Running, Role: LEADER
I20240503 07:59:29.006114 929 consensus_queue.cc:237] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec08b404c1f45f5bdb996289b593b78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44577 } }
I20240503 07:59:29.006400 492 catalog_manager.cc:5484] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c96d1d521954c79bd4840dc14893a37 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "8c96d1d521954c79bd4840dc14893a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.007606 917 tablet_bootstrap.cc:654] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.008908 917 tablet_bootstrap.cc:492] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: No bootstrap required, opened a new log
I20240503 07:59:29.008991 917 ts_tablet_manager.cc:1367] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:29.009337 917 raft_consensus.cc:348] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.009433 917 raft_consensus.cc:374] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.009478 917 raft_consensus.cc:729] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c96d1d521954c79bd4840dc14893a37, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.009624 917 consensus_queue.cc:260] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.009716 917 raft_consensus.cc:388] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.009748 917 raft_consensus.cc:482] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.009788 917 raft_consensus.cc:3037] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.010679 917 raft_consensus.cc:504] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.010655 491 catalog_manager.cc:5484] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ec08b404c1f45f5bdb996289b593b78 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "9ec08b404c1f45f5bdb996289b593b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec08b404c1f45f5bdb996289b593b78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44577 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.010808 917 leader_election.cc:302] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c96d1d521954c79bd4840dc14893a37; no voters:
I20240503 07:59:29.010883 917 leader_election.cc:288] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.011075 917 ts_tablet_manager.cc:1398] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:29.012132 927 raft_consensus.cc:2781] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.012319 927 raft_consensus.cc:686] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 1 LEADER]: Becoming Leader. State: Replica: 8c96d1d521954c79bd4840dc14893a37, State: Running, Role: LEADER
I20240503 07:59:29.012444 927 consensus_queue.cc:237] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } }
I20240503 07:59:29.015197 923 raft_consensus.cc:2781] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.016259 923 raft_consensus.cc:686] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 1 LEADER]: Becoming Leader. State: Replica: a48f9180ceed4835baadaae5bfd6f116, State: Running, Role: LEADER
I20240503 07:59:29.016273 492 catalog_manager.cc:5484] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c96d1d521954c79bd4840dc14893a37 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "8c96d1d521954c79bd4840dc14893a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96d1d521954c79bd4840dc14893a37" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39307 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.016410 923 consensus_queue.cc:237] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f9180ceed4835baadaae5bfd6f116" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44793 } }
I20240503 07:59:29.018361 491 catalog_manager.cc:5484] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 reported cstate change: term changed from 0 to 1, leader changed from <none> to a48f9180ceed4835baadaae5bfd6f116 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a48f9180ceed4835baadaae5bfd6f116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f9180ceed4835baadaae5bfd6f116" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44793 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.029649 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 07:59:29.034904 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:29.035185 396 tablet_replica.cc:330] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37: stopping tablet replica
I20240503 07:59:29.035459 396 raft_consensus.cc:2229] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.035624 396 raft_consensus.cc:2258] T 64f0cee18c444e3da5965dc24c05f8c4 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.037284 396 tablet_replica.cc:330] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37: stopping tablet replica
I20240503 07:59:29.037397 396 raft_consensus.cc:2229] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.037503 396 raft_consensus.cc:2258] T 13a1a265d0c14450b83661c705bb0443 P 8c96d1d521954c79bd4840dc14893a37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.041530 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 07:59:29.045405 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 07:59:29.049193 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:29.049427 396 tablet_replica.cc:330] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78: stopping tablet replica
I20240503 07:59:29.049538 396 raft_consensus.cc:2229] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.049657 396 raft_consensus.cc:2258] T 104f4fd670784a1aa61fab763cadbaf9 P 9ec08b404c1f45f5bdb996289b593b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.062885 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 07:59:29.064873 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 07:59:29.069494 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:29.069792 396 tablet_replica.cc:330] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116: stopping tablet replica
I20240503 07:59:29.069931 396 raft_consensus.cc:2229] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.070046 396 raft_consensus.cc:2258] T fa59484a1caa40a6b1ce21caad45c2ee P a48f9180ceed4835baadaae5bfd6f116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.083173 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 07:59:29.085060 396 master.cc:537] Master@127.0.99.62:43861 shutting down...
I20240503 07:59:29.088825 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.088982 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.089058 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 417af0034cab4f96ab54b49605bd884e: stopping tablet replica
I20240503 07:59:29.116277 396 master.cc:552] Master@127.0.99.62:43861 shutdown complete.
I20240503 07:59:29.119251 396 master.cc:537] Master@127.0.99.61:38161 shutting down...
I20240503 07:59:29.122153 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.122428 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.122488 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8f7f2f0f7e414fbca47b19678bf04f00: stopping tablet replica
I20240503 07:59:29.135637 396 master.cc:552] Master@127.0.99.61:38161 shutdown complete.
I20240503 07:59:29.138617 396 master.cc:537] Master@127.0.99.60:40345 shutting down...
I20240503 07:59:29.141942 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.142119 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.142191 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f65fde336c04151b3b1fd8f23744e4d: stopping tablet replica
I20240503 07:59:29.154891 396 master.cc:552] Master@127.0.99.60:40345 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2647 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 07:59:29.162859 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:45479
I20240503 07:59:29.163192 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.166025 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.166858 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.167654 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.168411 396 server_base.cc:1034] running on GCE node
I20240503 07:59:29.168571 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:29.168610 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:29.168625 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723169168625 us; error 0 us; skew 500 ppm
I20240503 07:59:29.169546 396 webserver.cc:460] Webserver started at http://127.0.99.62:37469/ using document root <none> and password file <none>
I20240503 07:59:29.169692 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.169875 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.169962 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.170387 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "d75d2657d7e349c6b4e7946d92b05906"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.171829 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:29.172809 949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.173099 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.173159 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "d75d2657d7e349c6b4e7946d92b05906"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.173231 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.180749 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.181027 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.185043 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:45479
I20240503 07:59:29.187481 1001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:45479 every 8 connection(s)
I20240503 07:59:29.187889 1002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.189563 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906: Bootstrap starting.
I20240503 07:59:29.190255 1002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.191140 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906: No bootstrap required, opened a new log
I20240503 07:59:29.191462 1002 raft_consensus.cc:348] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER }
I20240503 07:59:29.191545 1002 raft_consensus.cc:374] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.191565 1002 raft_consensus.cc:729] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75d2657d7e349c6b4e7946d92b05906, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.191671 1002 consensus_queue.cc:260] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER }
I20240503 07:59:29.191747 1002 raft_consensus.cc:388] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.191776 1002 raft_consensus.cc:482] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.191810 1002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.192456 1002 raft_consensus.cc:504] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER }
I20240503 07:59:29.192559 1002 leader_election.cc:302] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75d2657d7e349c6b4e7946d92b05906; no voters:
I20240503 07:59:29.192688 1002 leader_election.cc:288] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.192814 1005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.193004 1005 raft_consensus.cc:686] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 1 LEADER]: Becoming Leader. State: Replica: d75d2657d7e349c6b4e7946d92b05906, State: Running, Role: LEADER
I20240503 07:59:29.193131 1005 consensus_queue.cc:237] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER }
I20240503 07:59:29.193133 1002 sys_catalog.cc:564] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:29.193567 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d75d2657d7e349c6b4e7946d92b05906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER } }
I20240503 07:59:29.193660 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:29.193614 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d75d2657d7e349c6b4e7946d92b05906. Latest consensus state: current_term: 1 leader_uuid: "d75d2657d7e349c6b4e7946d92b05906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75d2657d7e349c6b4e7946d92b05906" member_type: VOTER } }
I20240503 07:59:29.193703 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:29.193934 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:29.194833 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:29.195083 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 07:59:29.196547 1009 catalog_manager.cc:1347] Generated new cluster ID: c766dffb946149bba36662bca5a83cea
I20240503 07:59:29.196593 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:29.225472 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:29.226043 1009 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:29.237622 1009 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906: Generated new TSK 0
I20240503 07:59:29.237794 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:29.259788 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.262830 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.263702 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.264479 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.265370 396 server_base.cc:1034] running on GCE node
I20240503 07:59:29.265568 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:29.265614 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:29.265642 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723169265641 us; error 0 us; skew 500 ppm
I20240503 07:59:29.266628 396 webserver.cc:460] Webserver started at http://127.0.99.1:45189/ using document root <none> and password file <none>
I20240503 07:59:29.266816 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.266870 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.266954 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.267349 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.268918 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:29.269973 1031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.270301 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.270367 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.270429 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.279448 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.279747 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.279968 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.280449 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.280488 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.280517 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.280546 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.284296 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43299
I20240503 07:59:29.284335 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43299 every 8 connection(s)
I20240503 07:59:29.285065 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.287914 1099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.288753 1102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.289535 396 server_base.cc:1034] running on GCE node
I20240503 07:59:29.289657 1095 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45479
I20240503 07:59:29.289757 1095 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.289973 1095 heartbeater.cc:511] Master 127.0.99.62:45479 requested a full tablet report, sending...
W20240503 07:59:29.290270 1100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.290503 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:29.290572 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:29.290623 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723169290622 us; error 0 us; skew 500 ppm
I20240503 07:59:29.290692 966 ts_manager.cc:194] Registered new tserver with Master: 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299)
I20240503 07:59:29.291554 396 webserver.cc:460] Webserver started at http://127.0.99.2:44357/ using document root <none> and password file <none>
I20240503 07:59:29.291687 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49358
I20240503 07:59:29.291707 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.291782 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.291850 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.292312 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.294275 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:29.295377 1107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.295639 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.295719 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.295792 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.305681 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.306059 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.306340 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.306769 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.306807 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.306859 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.306890 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.310791 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:36683
I20240503 07:59:29.310828 1170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:36683 every 8 connection(s)
I20240503 07:59:29.311589 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.314360 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.315120 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.315500 1171 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45479
I20240503 07:59:29.315598 1171 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.315786 1171 heartbeater.cc:511] Master 127.0.99.62:45479 requested a full tablet report, sending...
W20240503 07:59:29.315868 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.316358 966 ts_manager.cc:194] Registered new tserver with Master: 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683)
I20240503 07:59:29.316629 396 server_base.cc:1034] running on GCE node
I20240503 07:59:29.316839 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:29.316886 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:29.316905 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723169316905 us; error 0 us; skew 500 ppm
I20240503 07:59:29.317024 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49366
I20240503 07:59:29.317976 396 webserver.cc:460] Webserver started at http://127.0.99.3:39781/ using document root <none> and password file <none>
I20240503 07:59:29.318171 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.318229 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.318305 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.318686 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "c7bd064c74e04408a52c51fd928190ac"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.320116 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:29.321058 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.321389 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.321478 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "c7bd064c74e04408a52c51fd928190ac"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-mxwh"
I20240503 07:59:29.321556 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.341629 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.342056 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.342377 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.342837 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.342885 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.342934 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.342967 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.347623 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:44837
I20240503 07:59:29.347697 1245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:44837 every 8 connection(s)
I20240503 07:59:29.351917 1246 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45479
I20240503 07:59:29.352013 1246 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.352299 1246 heartbeater.cc:511] Master 127.0.99.62:45479 requested a full tablet report, sending...
I20240503 07:59:29.352922 966 ts_manager.cc:194] Registered new tserver with Master: c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:29.353469 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49378
I20240503 07:59:29.353643 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005630393s
I20240503 07:59:30.292733 1095 heartbeater.cc:503] Master 127.0.99.62:45479 was elected leader, sending a full tablet report...
I20240503 07:59:30.317950 1171 heartbeater.cc:503] Master 127.0.99.62:45479 was elected leader, sending a full tablet report...
I20240503 07:59:30.354446 1246 heartbeater.cc:503] Master 127.0.99.62:45479 was elected leader, sending a full tablet report...
I20240503 07:59:30.379855 396 test_util.cc:268] Using random seed: 427124812
I20240503 07:59:30.389073 966 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49390:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 07:59:30.389700 966 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 07:59:30.403426 1210 tablet_service.cc:1467] Processing CreateTablet for tablet 60084043fac34a80824ba17d6de51a1e (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 07:59:30.403782 1058 tablet_service.cc:1467] Processing CreateTablet for tablet f7744b889d594a6d889458953ba076f0 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 07:59:30.404032 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7744b889d594a6d889458953ba076f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.403985 1209 tablet_service.cc:1467] Processing CreateTablet for tablet f7744b889d594a6d889458953ba076f0 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 07:59:30.404107 1059 tablet_service.cc:1467] Processing CreateTablet for tablet 60084043fac34a80824ba17d6de51a1e (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 07:59:30.404315 1133 tablet_service.cc:1467] Processing CreateTablet for tablet b8055fd8d20b43f58f5905a95f2fd852 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 07:59:30.404392 1208 tablet_service.cc:1467] Processing CreateTablet for tablet b8055fd8d20b43f58f5905a95f2fd852 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 07:59:30.404531 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8055fd8d20b43f58f5905a95f2fd852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.404618 1057 tablet_service.cc:1467] Processing CreateTablet for tablet b8055fd8d20b43f58f5905a95f2fd852 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 07:59:30.404826 1207 tablet_service.cc:1467] Processing CreateTablet for tablet 640c175aff704ac18a44309471d83d69 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 07:59:30.404985 1056 tablet_service.cc:1467] Processing CreateTablet for tablet 640c175aff704ac18a44309471d83d69 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 07:59:30.405252 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 07:59:30.405352 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 07:59:30.405659 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 17c0ea2987984a77800ec0f6c6a0feed (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 07:59:30.405761 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 17c0ea2987984a77800ec0f6c6a0feed (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 07:59:30.406103 1204 tablet_service.cc:1467] Processing CreateTablet for tablet e4a702d8d6494fc88cd3738f29ec29e3 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 07:59:30.406143 1053 tablet_service.cc:1467] Processing CreateTablet for tablet e4a702d8d6494fc88cd3738f29ec29e3 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 07:59:30.406514 1203 tablet_service.cc:1467] Processing CreateTablet for tablet 3301d2ca649a461a97575c9d15686905 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 07:59:30.406499 1052 tablet_service.cc:1467] Processing CreateTablet for tablet 3301d2ca649a461a97575c9d15686905 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 07:59:30.404107 1135 tablet_service.cc:1467] Processing CreateTablet for tablet 60084043fac34a80824ba17d6de51a1e (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 07:59:30.407007 1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60084043fac34a80824ba17d6de51a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.404392 1134 tablet_service.cc:1467] Processing CreateTablet for tablet f7744b889d594a6d889458953ba076f0 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 07:59:30.407315 1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7744b889d594a6d889458953ba076f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.408385 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 640c175aff704ac18a44309471d83d69 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 07:59:30.408608 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c175aff704ac18a44309471d83d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.404392 1059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60084043fac34a80824ba17d6de51a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.409354 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.409498 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c0ea2987984a77800ec0f6c6a0feed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.403788 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60084043fac34a80824ba17d6de51a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.410218 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 07:59:30.410454 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.411638 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c175aff704ac18a44309471d83d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.411692 1266 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.412281 1268 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.412447 1130 tablet_service.cc:1467] Processing CreateTablet for tablet 17c0ea2987984a77800ec0f6c6a0feed (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 07:59:30.412705 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c0ea2987984a77800ec0f6c6a0feed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.411644 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab778f8d74d4f50bd7c3eceb9d29b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.413722 1266 tablet_bootstrap.cc:654] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.413971 1209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7744b889d594a6d889458953ba076f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.414799 1129 tablet_service.cc:1467] Processing CreateTablet for tablet e4a702d8d6494fc88cd3738f29ec29e3 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 07:59:30.415041 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a702d8d6494fc88cd3738f29ec29e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.415468 1057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8055fd8d20b43f58f5905a95f2fd852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.415675 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8055fd8d20b43f58f5905a95f2fd852. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.416301 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3301d2ca649a461a97575c9d15686905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.417125 1128 tablet_service.cc:1467] Processing CreateTablet for tablet 3301d2ca649a461a97575c9d15686905 (DEFAULT_TABLE table=test-workload [id=fb45acb020a64124801d62e995091563]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 07:59:30.417347 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a702d8d6494fc88cd3738f29ec29e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.418095 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c0ea2987984a77800ec0f6c6a0feed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.418782 1266 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.418864 1266 ts_tablet_manager.cc:1367] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:30.417344 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3301d2ca649a461a97575c9d15686905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.419319 1266 raft_consensus.cc:348] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.419816 1266 raft_consensus.cc:374] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.419950 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a702d8d6494fc88cd3738f29ec29e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.419852 1266 raft_consensus.cc:729] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.420245 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c175aff704ac18a44309471d83d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.420269 1266 consensus_queue.cc:260] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.420655 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3301d2ca649a461a97575c9d15686905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.420652 1266 ts_tablet_manager.cc:1398] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.420864 1266 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.421880 1266 tablet_bootstrap.cc:654] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.425375 1268 tablet_bootstrap.cc:654] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.425716 1266 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.425807 1266 ts_tablet_manager.cc:1367] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:30.426154 1267 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.426237 1266 raft_consensus.cc:348] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.426349 1266 raft_consensus.cc:374] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.426388 1266 raft_consensus.cc:729] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.426524 1266 consensus_queue.cc:260] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.426765 1266 ts_tablet_manager.cc:1398] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.426916 1266 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.427114 1267 tablet_bootstrap.cc:654] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.427945 1266 tablet_bootstrap.cc:654] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.429004 1268 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.429091 1266 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.429121 1268 ts_tablet_manager.cc:1367] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:30.429193 1266 ts_tablet_manager.cc:1367] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:30.429622 1266 raft_consensus.cc:348] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.429721 1266 raft_consensus.cc:374] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.429678 1268 raft_consensus.cc:348] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.429757 1266 raft_consensus.cc:729] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.429786 1268 raft_consensus.cc:374] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.429821 1268 raft_consensus.cc:729] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.429872 1266 consensus_queue.cc:260] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.429950 1268 consensus_queue.cc:260] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.430059 1267 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.430138 1267 ts_tablet_manager.cc:1367] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.430161 1266 ts_tablet_manager.cc:1398] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:30.430302 1268 ts_tablet_manager.cc:1398] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.430351 1266 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.430451 1268 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.430513 1267 raft_consensus.cc:348] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.430588 1267 raft_consensus.cc:374] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.430624 1267 raft_consensus.cc:729] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.430738 1267 consensus_queue.cc:260] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.431079 1267 ts_tablet_manager.cc:1398] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.431226 1267 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.431313 1266 tablet_bootstrap.cc:654] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.431383 1268 tablet_bootstrap.cc:654] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.432143 1267 tablet_bootstrap.cc:654] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.433657 1267 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.433750 1266 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.433910 1266 ts_tablet_manager.cc:1367] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:30.434098 1268 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.434199 1268 ts_tablet_manager.cc:1367] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.434480 1266 raft_consensus.cc:348] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.434585 1266 raft_consensus.cc:374] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.434625 1266 raft_consensus.cc:729] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.434641 1268 raft_consensus.cc:348] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.434759 1268 raft_consensus.cc:374] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.434834 1268 raft_consensus.cc:729] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.434757 1266 consensus_queue.cc:260] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.434962 1268 consensus_queue.cc:260] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.435077 1266 ts_tablet_manager.cc:1398] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.435257 1268 ts_tablet_manager.cc:1398] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.435283 1266 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.435417 1268 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.436214 1266 tablet_bootstrap.cc:654] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.436305 1268 tablet_bootstrap.cc:654] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.433748 1267 ts_tablet_manager.cc:1367] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:30.437268 1267 raft_consensus.cc:348] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.437371 1267 raft_consensus.cc:374] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.437410 1267 raft_consensus.cc:729] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.437484 1266 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.437567 1266 ts_tablet_manager.cc:1367] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:30.437520 1267 consensus_queue.cc:260] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.437808 1267 ts_tablet_manager.cc:1398] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.437908 1266 raft_consensus.cc:348] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.437981 1267 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.438009 1266 raft_consensus.cc:374] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.438045 1266 raft_consensus.cc:729] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.438195 1266 consensus_queue.cc:260] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.438555 1266 ts_tablet_manager.cc:1398] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.438705 1266 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.438884 1267 tablet_bootstrap.cc:654] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.439615 1266 tablet_bootstrap.cc:654] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.440694 1268 tablet_bootstrap.cc:492] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.440800 1268 ts_tablet_manager.cc:1367] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:30.440825 1267 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.440940 1267 ts_tablet_manager.cc:1367] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:30.441198 1268 raft_consensus.cc:348] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.441275 1268 raft_consensus.cc:374] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.441320 1268 raft_consensus.cc:729] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.441408 1267 raft_consensus.cc:348] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.441433 1268 consensus_queue.cc:260] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.441532 1267 raft_consensus.cc:374] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.441592 1267 raft_consensus.cc:729] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.441720 1267 consensus_queue.cc:260] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.441815 1268 ts_tablet_manager.cc:1398] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.441964 1267 ts_tablet_manager.cc:1398] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.442008 1266 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.442004 1268 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.442107 1266 ts_tablet_manager.cc:1367] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:30.442507 1266 raft_consensus.cc:348] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.442616 1266 raft_consensus.cc:374] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.442653 1266 raft_consensus.cc:729] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.442792 1266 consensus_queue.cc:260] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.442978 1268 tablet_bootstrap.cc:654] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.443615 1266 ts_tablet_manager.cc:1398] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:30.443768 1266 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.444550 1266 tablet_bootstrap.cc:654] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.444821 1268 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.444907 1268 ts_tablet_manager.cc:1367] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:30.442107 1267 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.445331 1268 raft_consensus.cc:348] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.445432 1268 raft_consensus.cc:374] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.445479 1268 raft_consensus.cc:729] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.445612 1268 consensus_queue.cc:260] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.445931 1268 ts_tablet_manager.cc:1398] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.445968 1267 tablet_bootstrap.cc:654] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.446125 1268 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.446943 1268 tablet_bootstrap.cc:654] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.447129 1267 tablet_bootstrap.cc:492] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.447216 1267 ts_tablet_manager.cc:1367] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:30.447710 1267 raft_consensus.cc:348] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.447813 1267 raft_consensus.cc:374] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.447846 1267 raft_consensus.cc:729] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.447937 1266 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.448009 1266 ts_tablet_manager.cc:1367] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:30.447973 1267 consensus_queue.cc:260] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.448042 1268 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.448112 1268 ts_tablet_manager.cc:1367] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:30.448412 1266 raft_consensus.cc:348] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.448527 1266 raft_consensus.cc:374] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.448503 1268 raft_consensus.cc:348] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.448606 1268 raft_consensus.cc:374] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.448642 1268 raft_consensus.cc:729] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.448742 1268 consensus_queue.cc:260] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.449043 1268 ts_tablet_manager.cc:1398] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.449229 1268 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.450083 1267 ts_tablet_manager.cc:1398] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:30.450136 1268 tablet_bootstrap.cc:654] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.450268 1267 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.448559 1266 raft_consensus.cc:729] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.450723 1266 consensus_queue.cc:260] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.450968 1266 ts_tablet_manager.cc:1398] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:30.451151 1267 tablet_bootstrap.cc:654] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.451153 1266 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: Bootstrap starting.
I20240503 07:59:30.451997 1266 tablet_bootstrap.cc:654] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.452378 1267 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.452474 1267 ts_tablet_manager.cc:1367] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:30.452924 1267 raft_consensus.cc:348] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.453024 1267 raft_consensus.cc:374] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.453058 1267 raft_consensus.cc:729] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.453079 1268 tablet_bootstrap.cc:492] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.453150 1268 ts_tablet_manager.cc:1367] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:30.453189 1267 consensus_queue.cc:260] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.453341 1266 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: No bootstrap required, opened a new log
I20240503 07:59:30.453418 1266 ts_tablet_manager.cc:1367] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:30.453531 1268 raft_consensus.cc:348] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.453636 1268 raft_consensus.cc:374] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.453675 1268 raft_consensus.cc:729] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.453791 1268 consensus_queue.cc:260] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.453845 1266 raft_consensus.cc:348] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.453944 1266 raft_consensus.cc:374] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.453980 1266 raft_consensus.cc:729] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.454037 1268 ts_tablet_manager.cc:1398] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:30.454027 1267 ts_tablet_manager.cc:1398] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.454113 1266 consensus_queue.cc:260] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.454226 1268 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.454224 1267 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.454407 1266 ts_tablet_manager.cc:1398] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.455138 1268 tablet_bootstrap.cc:654] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.455252 1270 raft_consensus.cc:482] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.455339 1270 raft_consensus.cc:504] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.455138 1267 tablet_bootstrap.cc:654] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.455840 1270 leader_election.cc:288] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.459991 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c175aff704ac18a44309471d83d69" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" is_pre_election: true
I20240503 07:59:30.460170 1069 raft_consensus.cc:2454] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 0.
I20240503 07:59:30.460574 1108 leader_election.cc:302] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.460881 1270 raft_consensus.cc:2781] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.460952 1270 raft_consensus.cc:482] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.460999 1270 raft_consensus.cc:3037] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.461146 1267 tablet_bootstrap.cc:492] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.461227 1267 ts_tablet_manager.cc:1367] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:30.461688 1267 raft_consensus.cc:348] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.461799 1267 raft_consensus.cc:374] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.461836 1267 raft_consensus.cc:729] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.461947 1267 consensus_queue.cc:260] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.462221 1267 ts_tablet_manager.cc:1398] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.462450 1267 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.463553 1268 tablet_bootstrap.cc:492] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.463632 1268 ts_tablet_manager.cc:1367] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:30.463615 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c175aff704ac18a44309471d83d69" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac" is_pre_election: true
I20240503 07:59:30.463901 1267 tablet_bootstrap.cc:654] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.464079 1268 raft_consensus.cc:348] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.464196 1268 raft_consensus.cc:374] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.464237 1268 raft_consensus.cc:729] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.464355 1268 consensus_queue.cc:260] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
W20240503 07:59:30.464385 1109 leader_election.cc:341] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:30.464768 1268 ts_tablet_manager.cc:1398] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.464944 1268 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: Bootstrap starting.
I20240503 07:59:30.465543 1267 tablet_bootstrap.cc:492] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.465637 1267 ts_tablet_manager.cc:1367] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:30.465978 1268 tablet_bootstrap.cc:654] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.466082 1267 raft_consensus.cc:348] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.466177 1267 raft_consensus.cc:374] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.466214 1267 raft_consensus.cc:729] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.466333 1267 consensus_queue.cc:260] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.466697 1267 ts_tablet_manager.cc:1398] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.466866 1267 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Bootstrap starting.
I20240503 07:59:30.467412 1268 tablet_bootstrap.cc:492] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: No bootstrap required, opened a new log
I20240503 07:59:30.467494 1268 ts_tablet_manager.cc:1367] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:30.467823 1268 raft_consensus.cc:348] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.467913 1268 raft_consensus.cc:374] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.467948 1268 raft_consensus.cc:729] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.468075 1268 consensus_queue.cc:260] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.468235 1267 tablet_bootstrap.cc:654] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.468379 1268 ts_tablet_manager.cc:1398] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.468744 1275 raft_consensus.cc:482] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.468830 1275 raft_consensus.cc:504] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.469369 1275 leader_election.cc:288] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683)
I20240503 07:59:30.470685 1267 tablet_bootstrap.cc:492] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: No bootstrap required, opened a new log
I20240503 07:59:30.470788 1267 ts_tablet_manager.cc:1367] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.471213 1267 raft_consensus.cc:348] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.471753 1267 raft_consensus.cc:374] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.471791 1267 raft_consensus.cc:729] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.471891 1267 consensus_queue.cc:260] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.472148 1267 ts_tablet_manager.cc:1398] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.473495 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7744b889d594a6d889458953ba076f0" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" is_pre_election: true
I20240503 07:59:30.473636 1069 raft_consensus.cc:2454] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.473902 1183 leader_election.cc:302] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.473945 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7744b889d594a6d889458953ba076f0" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.474056 1145 raft_consensus.cc:2454] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.474071 1275 raft_consensus.cc:2781] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.474118 1275 raft_consensus.cc:482] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.474155 1275 raft_consensus.cc:3037] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.489084 1276 raft_consensus.cc:482] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.489231 1276 raft_consensus.cc:504] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.489753 1276 leader_election.cc:288] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.490760 1276 raft_consensus.cc:482] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.490861 1276 raft_consensus.cc:504] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.491425 1276 leader_election.cc:288] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.493408 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8055fd8d20b43f58f5905a95f2fd852" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.493458 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab778f8d74d4f50bd7c3eceb9d29b93" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.493543 1144 raft_consensus.cc:2454] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.493543 1145 raft_consensus.cc:2454] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.493669 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8055fd8d20b43f58f5905a95f2fd852" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac" is_pre_election: true
I20240503 07:59:30.493814 1034 leader_election.cc:302] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.493818 1220 raft_consensus.cc:2454] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.493814 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab778f8d74d4f50bd7c3eceb9d29b93" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac" is_pre_election: true
I20240503 07:59:30.493991 1276 raft_consensus.cc:2781] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.494019 1034 leader_election.cc:302] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.494040 1276 raft_consensus.cc:482] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.494045 1219 raft_consensus.cc:2454] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.494102 1276 raft_consensus.cc:3037] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.494434 1302 raft_consensus.cc:2781] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.494493 1302 raft_consensus.cc:482] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.494534 1302 raft_consensus.cc:3037] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.556982 1276 raft_consensus.cc:504] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.556982 1270 raft_consensus.cc:504] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.557343 1302 raft_consensus.cc:504] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.556982 1275 raft_consensus.cc:504] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.557605 1270 leader_election.cc:288] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 election: Requested vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.557605 1276 leader_election.cc:288] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Requested vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.557602 1302 leader_election.cc:288] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Requested vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.557889 1275 leader_election.cc:288] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Requested vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683)
I20240503 07:59:30.558040 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c175aff704ac18a44309471d83d69" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac"
I20240503 07:59:30.558157 1219 raft_consensus.cc:3037] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.558300 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c175aff704ac18a44309471d83d69" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
I20240503 07:59:30.558413 1069 raft_consensus.cc:3037] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.559211 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab778f8d74d4f50bd7c3eceb9d29b93" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.559326 1144 raft_consensus.cc:3037] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.559458 1069 raft_consensus.cc:2454] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 1.
I20240503 07:59:30.559479 1219 raft_consensus.cc:2454] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 1.
I20240503 07:59:30.559798 1109 leader_election.cc:302] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb3eebba53f465793dbf50c7a6ddafe, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.559809 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7744b889d594a6d889458953ba076f0" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.560045 1143 raft_consensus.cc:3037] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.560102 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8055fd8d20b43f58f5905a95f2fd852" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.560245 1145 raft_consensus.cc:3037] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.560251 1144 raft_consensus.cc:2454] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.560760 1034 leader_election.cc:302] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.560983 1302 raft_consensus.cc:2781] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.561195 1145 raft_consensus.cc:2454] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.561220 1302 raft_consensus.cc:686] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Becoming Leader. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Running, Role: LEADER
I20240503 07:59:30.561390 1270 raft_consensus.cc:2781] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.561473 1034 leader_election.cc:302] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.561583 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab778f8d74d4f50bd7c3eceb9d29b93" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac"
I20240503 07:59:30.561645 1276 raft_consensus.cc:2781] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.561515 1302 consensus_queue.cc:237] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.561288 1143 raft_consensus.cc:2454] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.561755 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8055fd8d20b43f58f5905a95f2fd852" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac"
I20240503 07:59:30.561614 1270 raft_consensus.cc:686] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Becoming Leader. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Running, Role: LEADER
I20240503 07:59:30.561818 1276 raft_consensus.cc:686] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Becoming Leader. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Running, Role: LEADER
I20240503 07:59:30.561712 1218 raft_consensus.cc:3037] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.561918 1270 consensus_queue.cc:237] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.561956 1276 consensus_queue.cc:237] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.561829 1220 raft_consensus.cc:3037] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.562705 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7744b889d594a6d889458953ba076f0" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
I20240503 07:59:30.562794 1069 raft_consensus.cc:3037] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.563311 1185 leader_election.cc:302] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb3eebba53f465793dbf50c7a6ddafe, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.563488 1275 raft_consensus.cc:2781] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.563652 1275 raft_consensus.cc:686] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Becoming Leader. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Running, Role: LEADER
I20240503 07:59:30.563771 1275 consensus_queue.cc:237] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.563866 966 catalog_manager.cc:5484] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2). New cstate: current_term: 1 leader_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.564469 1275 raft_consensus.cc:482] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.564536 1275 raft_consensus.cc:504] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.564641 963 catalog_manager.cc:5484] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.564851 1275 leader_election.cc:288] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683)
I20240503 07:59:30.565376 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3301d2ca649a461a97575c9d15686905" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" is_pre_election: true
I20240503 07:59:30.565521 1068 raft_consensus.cc:2454] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.565377 964 catalog_manager.cc:5484] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac reported cstate change: term changed from 0 to 1, leader changed from <none> to c7bd064c74e04408a52c51fd928190ac (127.0.99.3). New cstate: current_term: 1 leader_uuid: "c7bd064c74e04408a52c51fd928190ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.565757 1183 leader_election.cc:302] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.565925 1275 raft_consensus.cc:2781] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.565994 1275 raft_consensus.cc:482] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.566071 1275 raft_consensus.cc:3037] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.565937 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3301d2ca649a461a97575c9d15686905" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.566280 1143 raft_consensus.cc:2454] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.568053 964 catalog_manager.cc:5484] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.574702 1270 raft_consensus.cc:482] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.574826 1270 raft_consensus.cc:504] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.575177 1270 leader_election.cc:288] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.575537 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a702d8d6494fc88cd3738f29ec29e3" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" is_pre_election: true
I20240503 07:59:30.575534 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a702d8d6494fc88cd3738f29ec29e3" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac" is_pre_election: true
I20240503 07:59:30.575659 1068 raft_consensus.cc:2454] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 0.
I20240503 07:59:30.575678 1219 raft_consensus.cc:2454] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 0.
I20240503 07:59:30.575902 1109 leader_election.cc:302] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb3eebba53f465793dbf50c7a6ddafe, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.576052 1270 raft_consensus.cc:2781] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.576105 1270 raft_consensus.cc:482] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.576159 1270 raft_consensus.cc:3037] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.649138 1308 raft_consensus.cc:482] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.649288 1308 raft_consensus.cc:504] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.649775 1308 leader_election.cc:288] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299)
I20240503 07:59:30.650211 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60084043fac34a80824ba17d6de51a1e" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.650211 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60084043fac34a80824ba17d6de51a1e" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" is_pre_election: true
I20240503 07:59:30.650393 1068 raft_consensus.cc:2454] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.650393 1143 raft_consensus.cc:2454] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 0.
I20240503 07:59:30.650732 1185 leader_election.cc:302] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb3eebba53f465793dbf50c7a6ddafe, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.650866 1218 raft_consensus.cc:2454] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.650991 1308 raft_consensus.cc:2781] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.651036 1308 raft_consensus.cc:482] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.651085 1308 raft_consensus.cc:3037] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.651010 1270 raft_consensus.cc:504] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.651446 1270 leader_election.cc:288] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 election: Requested vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.651990 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a702d8d6494fc88cd3738f29ec29e3" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
I20240503 07:59:30.652032 1308 raft_consensus.cc:504] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.652112 1068 raft_consensus.cc:3037] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.652315 1220 raft_consensus.cc:2454] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.652383 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a702d8d6494fc88cd3738f29ec29e3" candidate_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac"
I20240503 07:59:30.652426 1308 leader_election.cc:288] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Requested vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299)
I20240503 07:59:30.652475 1218 raft_consensus.cc:3037] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.653221 1275 raft_consensus.cc:504] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.653456 1275 leader_election.cc:288] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Requested vote from peers 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1:43299), 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683)
I20240503 07:59:30.653514 1218 raft_consensus.cc:2454] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 1.
I20240503 07:59:30.653663 1069 raft_consensus.cc:2454] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.654202 1109 leader_election.cc:302] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb3eebba53f465793dbf50c7a6ddafe, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.654244 1067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3301d2ca649a461a97575c9d15686905" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
I20240503 07:59:30.654352 1067 raft_consensus.cc:3037] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.654433 1068 raft_consensus.cc:2454] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb3eebba53f465793dbf50c7a6ddafe in term 1.
I20240503 07:59:30.654724 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60084043fac34a80824ba17d6de51a1e" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.654834 1143 raft_consensus.cc:3037] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.655069 1067 raft_consensus.cc:2454] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.655256 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60084043fac34a80824ba17d6de51a1e" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd"
I20240503 07:59:30.655340 1069 raft_consensus.cc:3037] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.655655 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3301d2ca649a461a97575c9d15686905" candidate_uuid: "c7bd064c74e04408a52c51fd928190ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.655761 1145 raft_consensus.cc:3037] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.655789 1183 leader_election.cc:302] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.655934 1270 raft_consensus.cc:2781] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.655961 1275 raft_consensus.cc:2781] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.656018 1270 raft_consensus.cc:686] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Becoming Leader. State: Replica: 5cb3eebba53f465793dbf50c7a6ddafe, State: Running, Role: LEADER
I20240503 07:59:30.656054 1275 raft_consensus.cc:686] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Becoming Leader. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Running, Role: LEADER
I20240503 07:59:30.656213 1275 consensus_queue.cc:237] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.656445 1270 consensus_queue.cc:237] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.657799 964 catalog_manager.cc:5484] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac reported cstate change: term changed from 0 to 1, leader changed from <none> to c7bd064c74e04408a52c51fd928190ac (127.0.99.3). New cstate: current_term: 1 leader_uuid: "c7bd064c74e04408a52c51fd928190ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.658247 966 catalog_manager.cc:5484] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2). New cstate: current_term: 1 leader_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.740402 1302 raft_consensus.cc:482] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.740550 1302 raft_consensus.cc:504] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.740932 1302 leader_election.cc:288] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.741389 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c0ea2987984a77800ec0f6c6a0feed" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" is_pre_election: true
I20240503 07:59:30.741397 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c0ea2987984a77800ec0f6c6a0feed" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac" is_pre_election: true
I20240503 07:59:30.741510 1144 raft_consensus.cc:2454] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.741567 1218 raft_consensus.cc:2454] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 0.
I20240503 07:59:30.741791 1034 leader_election.cc:302] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.741935 1302 raft_consensus.cc:2781] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.741992 1302 raft_consensus.cc:482] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.742035 1302 raft_consensus.cc:3037] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.745585 1069 raft_consensus.cc:2454] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.745587 1143 raft_consensus.cc:2454] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.746163 1183 leader_election.cc:302] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, c7bd064c74e04408a52c51fd928190ac; no voters:
I20240503 07:59:30.746368 1275 raft_consensus.cc:2781] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.746528 1275 raft_consensus.cc:686] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Becoming Leader. State: Replica: c7bd064c74e04408a52c51fd928190ac, State: Running, Role: LEADER
I20240503 07:59:30.746552 1145 raft_consensus.cc:2454] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7bd064c74e04408a52c51fd928190ac in term 1.
I20240503 07:59:30.746675 1275 consensus_queue.cc:237] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } }
I20240503 07:59:30.746708 1302 raft_consensus.cc:504] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.747140 1302 leader_election.cc:288] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Requested vote from peers 5cb3eebba53f465793dbf50c7a6ddafe (127.0.99.2:36683), c7bd064c74e04408a52c51fd928190ac (127.0.99.3:44837)
I20240503 07:59:30.747391 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c0ea2987984a77800ec0f6c6a0feed" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb3eebba53f465793dbf50c7a6ddafe"
I20240503 07:59:30.747484 1145 raft_consensus.cc:3037] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.747710 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c0ea2987984a77800ec0f6c6a0feed" candidate_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd064c74e04408a52c51fd928190ac"
I20240503 07:59:30.747809 1218 raft_consensus.cc:3037] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.748288 1145 raft_consensus.cc:2454] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.748616 1034 leader_election.cc:302] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57cf114d32aa4ea1b26d4d3fe5c6eecd, 5cb3eebba53f465793dbf50c7a6ddafe; no voters:
I20240503 07:59:30.748642 964 catalog_manager.cc:5484] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac reported cstate change: term changed from 0 to 1, leader changed from <none> to c7bd064c74e04408a52c51fd928190ac (127.0.99.3). New cstate: current_term: 1 leader_uuid: "c7bd064c74e04408a52c51fd928190ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.748854 1218 raft_consensus.cc:2454] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cf114d32aa4ea1b26d4d3fe5c6eecd in term 1.
I20240503 07:59:30.749127 1302 raft_consensus.cc:2781] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.749230 1302 raft_consensus.cc:686] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Becoming Leader. State: Replica: 57cf114d32aa4ea1b26d4d3fe5c6eecd, State: Running, Role: LEADER
I20240503 07:59:30.749387 1302 consensus_queue.cc:237] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } }
I20240503 07:59:30.751148 964 catalog_manager.cc:5484] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 57cf114d32aa4ea1b26d4d3fe5c6eecd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.758796 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:30.761621 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.762423 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.763154 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:30.763929 396 server_base.cc:1034] running on GCE node
I20240503 07:59:30.764094 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:30.764133 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:30.764183 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723170764182 us; error 0 us; skew 500 ppm
I20240503 07:59:30.765281 396 webserver.cc:460] Webserver started at http://127.0.99.4:40999/ using document root <none> and password file <none>
I20240503 07:59:30.765419 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:30.765467 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:30.765542 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:30.765900 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-3-root/instance:
uuid: "024d26119407423db4db9e853dcad5bd"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-mxwh"
I20240503 07:59:30.767381 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:30.768297 1323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.768525 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.768594 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-3-root
uuid: "024d26119407423db4db9e853dcad5bd"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-mxwh"
I20240503 07:59:30.768664 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723166506157-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:30.797197 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:30.797591 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:30.797894 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:30.798338 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:30.798378 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.798429 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:30.798461 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.803601 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:34137
I20240503 07:59:30.803653 1386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:34137 every 8 connection(s)
I20240503 07:59:30.807934 1387 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45479
I20240503 07:59:30.808101 1387 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:30.808365 1387 heartbeater.cc:511] Master 127.0.99.62:45479 requested a full tablet report, sending...
I20240503 07:59:30.808946 964 ts_manager.cc:194] Registered new tserver with Master: 024d26119407423db4db9e853dcad5bd (127.0.99.4:34137)
I20240503 07:59:30.809461 964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49404
I20240503 07:59:30.961064 1275 consensus_queue.cc:1035] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.965047 1308 consensus_queue.cc:1035] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.000036 1302 consensus_queue.cc:1035] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.003686 1302 consensus_queue.cc:1035] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.044708 1302 consensus_queue.cc:1035] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.048460 1302 consensus_queue.cc:1035] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.111809 1270 consensus_queue.cc:1035] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.115166 1400 consensus_queue.cc:1035] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.148869 1275 consensus_queue.cc:1035] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.151829 1275 consensus_queue.cc:1035] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.160714 1302 consensus_queue.cc:1035] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.163437 1302 consensus_queue.cc:1035] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.171880 1403 consensus_queue.cc:1035] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.174880 1308 consensus_queue.cc:1035] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.195187 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:31.196717 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:31.196900 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:31.231556 1270 consensus_queue.cc:1035] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.234498 1303 consensus_queue.cc:1035] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.810297 1387 heartbeater.cc:503] Master 127.0.99.62:45479 was elected leader, sending a full tablet report...
I20240503 07:59:33.197232 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:33.197989 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:33.198053 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:35.198359 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:35.199193 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:35.199267 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:36.204327 1218 consensus_queue.cc:237] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } }
I20240503 07:59:36.205960 1143 raft_consensus.cc:1261] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Refusing update from remote peer c7bd064c74e04408a52c51fd928190ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.206214 1069 raft_consensus.cc:1261] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Refusing update from remote peer c7bd064c74e04408a52c51fd928190ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.206635 1417 consensus_queue.cc:1035] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.206902 1411 consensus_queue.cc:1035] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.208690 1411 raft_consensus.cc:2932] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.209075 1143 raft_consensus.cc:2932] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
W20240503 07:59:36.210528 1183 consensus_peers.cc:485] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac -> Peer 024d26119407423db4db9e853dcad5bd (127.0.99.4:34137): Couldn't send request to peer 024d26119407423db4db9e853dcad5bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60084043fac34a80824ba17d6de51a1e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:36.210780 1069 raft_consensus.cc:2932] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.211339 966 catalog_manager.cc:5484] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe reported cstate change: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "c7bd064c74e04408a52c51fd928190ac" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } attrs { replace: true } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.215448 1143 consensus_queue.cc:237] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } }
I20240503 07:59:36.216917 1218 raft_consensus.cc:1261] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Refusing update from remote peer 5cb3eebba53f465793dbf50c7a6ddafe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.217085 1069 raft_consensus.cc:1261] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Refusing update from remote peer 5cb3eebba53f465793dbf50c7a6ddafe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.217283 1270 consensus_queue.cc:1035] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.217451 1428 consensus_queue.cc:1035] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.219597 1428 raft_consensus.cc:2932] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.220259 1218 raft_consensus.cc:2932] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.221586 1069 raft_consensus.cc:2932] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.222398 963 catalog_manager.cc:5484] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe reported cstate change: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.222566 1143 consensus_queue.cc:237] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } }
I20240503 07:59:36.223901 1218 raft_consensus.cc:1261] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Refusing update from remote peer 5cb3eebba53f465793dbf50c7a6ddafe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.224347 1270 consensus_queue.cc:1035] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:36.225023 1108 consensus_peers.cc:485] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe -> Peer 024d26119407423db4db9e853dcad5bd (127.0.99.4:34137): Couldn't send request to peer 024d26119407423db4db9e853dcad5bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 640c175aff704ac18a44309471d83d69. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:36.225575 1108 consensus_peers.cc:485] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe -> Peer 024d26119407423db4db9e853dcad5bd (127.0.99.4:34137): Couldn't send request to peer 024d26119407423db4db9e853dcad5bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4a702d8d6494fc88cd3738f29ec29e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:36.225742 1069 raft_consensus.cc:1261] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Refusing update from remote peer 5cb3eebba53f465793dbf50c7a6ddafe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.226027 1270 consensus_queue.cc:1035] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.226893 1433 raft_consensus.cc:2932] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.227321 1218 raft_consensus.cc:2932] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.228029 1069 raft_consensus.cc:2932] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.229445 966 catalog_manager.cc:5484] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac reported cstate change: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.234702 1069 consensus_queue.cc:237] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } }
I20240503 07:59:36.236145 1143 raft_consensus.cc:1261] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Refusing update from remote peer 57cf114d32aa4ea1b26d4d3fe5c6eecd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.236145 1218 raft_consensus.cc:1261] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Refusing update from remote peer 57cf114d32aa4ea1b26d4d3fe5c6eecd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.236778 1438 consensus_queue.cc:1035] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.236956 1276 consensus_queue.cc:1035] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:36.239167 1439 raft_consensus.cc:2932] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.239526 1218 raft_consensus.cc:2932] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.240365 1143 raft_consensus.cc:2932] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
W20240503 07:59:36.241768 1032 consensus_peers.cc:485] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd -> Peer 024d26119407423db4db9e853dcad5bd (127.0.99.4:34137): Couldn't send request to peer 024d26119407423db4db9e853dcad5bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8055fd8d20b43f58f5905a95f2fd852. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:36.241989 963 catalog_manager.cc:5484] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac reported cstate change: config changed from index -1 to 2, NON_VOTER 024d26119407423db4db9e853dcad5bd (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "57cf114d32aa4ea1b26d4d3fe5c6eecd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43299 } } peers { permanent_uuid: "5cb3eebba53f465793dbf50c7a6ddafe" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36683 } } peers { permanent_uuid: "c7bd064c74e04408a52c51fd928190ac" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 44837 } attrs { replace: true } } peers { permanent_uuid: "024d26119407423db4db9e853dcad5bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34137 } attrs { promote: true } } }
I20240503 07:59:36.323530 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
W20240503 07:59:36.327432 950 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20240503 07:59:36.328083 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:36.328261 396 tablet_replica.cc:330] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.328406 396 raft_consensus.cc:2229] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 07:59:36.328416 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 07:59:36.328491 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 07:59:36.328574 396 raft_consensus.cc:2258] T 60084043fac34a80824ba17d6de51a1e P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.329248 396 tablet_replica.cc:330] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.329362 396 raft_consensus.cc:2229] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.329527 396 raft_consensus.cc:2258] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.329860 396 tablet_replica.cc:330] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.329963 396 raft_consensus.cc:2229] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.330050 396 raft_consensus.cc:2258] T f7744b889d594a6d889458953ba076f0 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.330345 396 tablet_replica.cc:330] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.330444 396 raft_consensus.cc:2229] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.330595 396 raft_consensus.cc:2258] T 17c0ea2987984a77800ec0f6c6a0feed P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.330902 396 tablet_replica.cc:330] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.330998 396 raft_consensus.cc:2229] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.331090 396 raft_consensus.cc:2258] T 640c175aff704ac18a44309471d83d69 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.331508 396 tablet_replica.cc:330] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.331588 396 raft_consensus.cc:2229] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.331663 396 raft_consensus.cc:2258] T e4a702d8d6494fc88cd3738f29ec29e3 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.332273 396 tablet_replica.cc:330] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.332350 396 raft_consensus.cc:2229] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.332547 396 raft_consensus.cc:2258] T b8055fd8d20b43f58f5905a95f2fd852 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.333038 396 tablet_replica.cc:330] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd: stopping tablet replica
I20240503 07:59:36.333137 396 raft_consensus.cc:2229] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.333227 396 raft_consensus.cc:2258] T 3301d2ca649a461a97575c9d15686905 P 57cf114d32aa4ea1b26d4d3fe5c6eecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.348312 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 07:59:36.350934 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
W20240503 07:59:36.353415 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 07:59:36.353554 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 07:59:36.355999 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 07:59:36.356127 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 07:59:36.357193 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:36.357342 396 tablet_replica.cc:330] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.357497 396 raft_consensus.cc:2229] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.357707 396 raft_consensus.cc:2258] T 640c175aff704ac18a44309471d83d69 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.358273 396 tablet_replica.cc:330] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.358384 396 raft_consensus.cc:2229] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.358492 396 raft_consensus.cc:2258] T 60084043fac34a80824ba17d6de51a1e P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.358997 396 tablet_replica.cc:330] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.359103 396 raft_consensus.cc:2229] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.359198 396 raft_consensus.cc:2258] T 3301d2ca649a461a97575c9d15686905 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.359504 396 tablet_replica.cc:330] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.359601 396 raft_consensus.cc:2229] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.359714 396 raft_consensus.cc:2258] T b8055fd8d20b43f58f5905a95f2fd852 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.360206 396 tablet_replica.cc:330] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.360307 396 raft_consensus.cc:2229] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.360508 396 raft_consensus.cc:2258] T e4a702d8d6494fc88cd3738f29ec29e3 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.360985 396 tablet_replica.cc:330] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.361084 396 raft_consensus.cc:2229] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.361184 396 raft_consensus.cc:2258] T f7744b889d594a6d889458953ba076f0 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.361521 396 tablet_replica.cc:330] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.361621 396 raft_consensus.cc:2229] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.361719 396 raft_consensus.cc:2258] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.362042 396 tablet_replica.cc:330] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe: stopping tablet replica
I20240503 07:59:36.362125 396 raft_consensus.cc:2229] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.362221 396 raft_consensus.cc:2258] T 17c0ea2987984a77800ec0f6c6a0feed P 5cb3eebba53f465793dbf50c7a6ddafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.367592 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 07:59:36.370627 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
W20240503 07:59:36.373488 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 07:59:36.373564 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 07:59:36.375618 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:36.375787 396 tablet_replica.cc:330] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.375916 396 raft_consensus.cc:2229] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.376025 396 raft_consensus.cc:2258] T e4a702d8d6494fc88cd3738f29ec29e3 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.376544 396 tablet_replica.cc:330] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.376631 396 raft_consensus.cc:2229] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.376713 396 raft_consensus.cc:2258] T 640c175aff704ac18a44309471d83d69 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.377064 396 tablet_replica.cc:330] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.377135 396 raft_consensus.cc:2229] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.377207 396 raft_consensus.cc:2258] T 6ab778f8d74d4f50bd7c3eceb9d29b93 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.377449 396 tablet_replica.cc:330] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.377516 396 raft_consensus.cc:2229] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.377581 396 raft_consensus.cc:2258] T 17c0ea2987984a77800ec0f6c6a0feed P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.377813 396 tablet_replica.cc:330] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.377878 396 raft_consensus.cc:2229] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.378034 396 raft_consensus.cc:2258] T 60084043fac34a80824ba17d6de51a1e P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.378454 396 tablet_replica.cc:330] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.378525 396 raft_consensus.cc:2229] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.378648 396 raft_consensus.cc:2258] T f7744b889d594a6d889458953ba076f0 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.378954 396 tablet_replica.cc:330] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.379024 396 raft_consensus.cc:2229] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.379091 396 raft_consensus.cc:2258] T b8055fd8d20b43f58f5905a95f2fd852 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.379611 396 tablet_replica.cc:330] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac: stopping tablet replica
I20240503 07:59:36.379685 396 raft_consensus.cc:2229] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.379814 396 raft_consensus.cc:2258] T 3301d2ca649a461a97575c9d15686905 P c7bd064c74e04408a52c51fd928190ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.393669 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 07:59:36.396345 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 07:59:36.400538 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:36.413659 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 07:59:36.415903 396 master.cc:537] Master@127.0.99.62:45479 shutting down...
I20240503 07:59:36.419765 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.419939 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.420003 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d75d2657d7e349c6b4e7946d92b05906: stopping tablet replica
I20240503 07:59:36.433069 396 master.cc:552] Master@127.0.99.62:45479 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7281 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 07:59:36.444482 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:46757,127.0.99.61:45211,127.0.99.60:35391
I20240503 07:59:36.444893 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:36.448302 396 server_base.cc:1034] running on GCE node
W20240503 07:59:36.449138 1444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.449998 1445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.450803 1447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.450999 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.451042 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.451057 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176451058 us; error 0 us; skew 500 ppm
I20240503 07:59:36.452037 396 webserver.cc:460] Webserver started at http://127.0.99.62:37923/ using document root <none> and password file <none>
I20240503 07:59:36.452256 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.452313 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.452371 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.452749 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "752af6ee40b44ac29c3964194201da17"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.454213 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:36.455140 1452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.455389 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.455458 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "752af6ee40b44ac29c3964194201da17"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.455528 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:36.475534 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.475916 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.479825 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:46757
I20240503 07:59:36.480829 1504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:46757 every 8 connection(s)
I20240503 07:59:36.481032 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:36.485977 1505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.487169 1505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
W20240503 07:59:36.487282 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.488220 1510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.489089 1508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.489974 396 server_base.cc:1034] running on GCE node
I20240503 07:59:36.490134 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.490193 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.490264 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176490263 us; error 0 us; skew 500 ppm
I20240503 07:59:36.491394 396 webserver.cc:460] Webserver started at http://127.0.99.61:32937/ using document root <none> and password file <none>
I20240503 07:59:36.491551 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.491601 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.491678 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.491899 1505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.492132 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root/instance:
uuid: "88105ab869a6409e8428dc40caaf9b2a"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
W20240503 07:59:36.493239 1505 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.99.61:45211: connect: Connection refused (error 111)
I20240503 07:59:36.493880 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.494921 1518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.495215 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.495285 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
uuid: "88105ab869a6409e8428dc40caaf9b2a"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.495348 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:36.505465 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.505838 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.510111 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:45211
I20240503 07:59:36.510160 1570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:45211 every 8 connection(s)
I20240503 07:59:36.510823 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:36.510989 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.512410 1571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
W20240503 07:59:36.513599 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.514611 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.515707 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.516279 396 server_base.cc:1034] running on GCE node
I20240503 07:59:36.516429 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.516469 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.516491 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176516490 us; error 0 us; skew 500 ppm
I20240503 07:59:36.516741 1571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.517498 396 webserver.cc:460] Webserver started at http://127.0.99.60:34071/ using document root <none> and password file <none>
I20240503 07:59:36.517625 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.517678 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.517765 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.518122 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root/instance:
uuid: "2dcb9ed8c8564302bb686ae2bef06a47"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.519800 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.520742 1571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.520871 1583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.521107 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.521215 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
uuid: "2dcb9ed8c8564302bb686ae2bef06a47"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.521297 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 07:59:36.521963 1571 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:35391: Network error: Client connection negotiation failed: client connection to 127.0.99.60:35391: connect: Connection refused (error 111)
I20240503 07:59:36.530594 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.530970 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.534834 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:35391
I20240503 07:59:36.534875 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:35391 every 8 connection(s)
I20240503 07:59:36.535274 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 07:59:36.535678 1637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.536787 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.541121 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.545182 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.546137 1571 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } attempt: 1
I20240503 07:59:36.550405 1505 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } attempt: 1
I20240503 07:59:36.551328 1571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: Bootstrap starting.
I20240503 07:59:36.551328 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: Bootstrap starting.
I20240503 07:59:36.552421 1637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.552421 1571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.553939 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: No bootstrap required, opened a new log
I20240503 07:59:36.554104 1571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: No bootstrap required, opened a new log
I20240503 07:59:36.554105 1505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
I20240503 07:59:36.554425 1637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.554517 1637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.554575 1637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dcb9ed8c8564302bb686ae2bef06a47, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.554594 1571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.554713 1571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.554761 1571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88105ab869a6409e8428dc40caaf9b2a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.554718 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.554885 1571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.555032 1644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.555117 1644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.555255 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:36.555384 1571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: configured and running, proceeding with master startup.
W20240503 07:59:36.557428 1661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:36.557507 1661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 07:59:36.558364 1667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:36.558420 1667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:59:36.558822 1645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.558925 1645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.559726 1505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: Bootstrap starting.
I20240503 07:59:36.560601 1505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.561590 1505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: No bootstrap required, opened a new log
I20240503 07:59:36.561942 1505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.562029 1505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.562052 1505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752af6ee40b44ac29c3964194201da17, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.562142 1505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.562453 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.562551 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.562646 1505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:36.564113 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 07:59:36.564191 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 07:59:36.564687 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:36.564805 1680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:36.564862 1680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 07:59:36.567528 1682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.568493 1681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.569644 396 server_base.cc:1034] running on GCE node
W20240503 07:59:36.570271 1684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.570526 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.570574 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.570591 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176570591 us; error 0 us; skew 500 ppm
I20240503 07:59:36.571545 396 webserver.cc:460] Webserver started at http://127.0.99.1:41989/ using document root <none> and password file <none>
I20240503 07:59:36.571681 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.571725 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.571779 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.572139 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "af22abc6a9a24fa4b45ff5e4a41098ca"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.573663 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.574700 1689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.574983 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.575058 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "af22abc6a9a24fa4b45ff5e4a41098ca"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.575129 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:36.588187 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.588721 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.589040 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:36.589562 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:36.589602 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.589653 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:36.589684 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.594321 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:40255
I20240503 07:59:36.594349 1752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:40255 every 8 connection(s)
I20240503 07:59:36.595379 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:36.598935 1761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:36.599848 1760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.600180 1754 heartbeater.cc:347] Connected to a master server at 127.0.99.60:35391
I20240503 07:59:36.600229 1753 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46757
I20240503 07:59:36.600318 1753 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.600318 1754 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.600600 1754 heartbeater.cc:511] Master 127.0.99.60:35391 requested a full tablet report, sending...
I20240503 07:59:36.600636 1753 heartbeater.cc:511] Master 127.0.99.62:46757 requested a full tablet report, sending...
I20240503 07:59:36.600836 1755 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45211
I20240503 07:59:36.600927 1755 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.601137 1755 heartbeater.cc:511] Master 127.0.99.61:45211 requested a full tablet report, sending...
W20240503 07:59:36.601138 1763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.601356 1601 ts_manager.cc:194] Registered new tserver with Master: af22abc6a9a24fa4b45ff5e4a41098ca (127.0.99.1:40255)
I20240503 07:59:36.601356 1469 ts_manager.cc:194] Registered new tserver with Master: af22abc6a9a24fa4b45ff5e4a41098ca (127.0.99.1:40255)
I20240503 07:59:36.601824 1535 ts_manager.cc:194] Registered new tserver with Master: af22abc6a9a24fa4b45ff5e4a41098ca (127.0.99.1:40255)
I20240503 07:59:36.602030 396 server_base.cc:1034] running on GCE node
I20240503 07:59:36.602224 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.602283 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.602299 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176602300 us; error 0 us; skew 500 ppm
I20240503 07:59:36.603179 396 webserver.cc:460] Webserver started at http://127.0.99.2:33069/ using document root <none> and password file <none>
I20240503 07:59:36.603307 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.603354 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.603412 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.603775 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "10517edc7ed743fc888ce2aa19fbb6b3"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.605273 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:36.606187 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.606416 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.606475 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "10517edc7ed743fc888ce2aa19fbb6b3"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.606526 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:36.621167 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.621594 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.621878 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:36.622373 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:36.622416 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.622473 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:36.622502 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.627059 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:39847
I20240503 07:59:36.627123 1831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:39847 every 8 connection(s)
I20240503 07:59:36.628041 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:36.631371 1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.632375 396 server_base.cc:1034] running on GCE node
I20240503 07:59:36.632484 1834 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45211
I20240503 07:59:36.632586 1834 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.632874 1834 heartbeater.cc:511] Master 127.0.99.61:45211 requested a full tablet report, sending...
W20240503 07:59:36.633273 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.633314 1833 heartbeater.cc:347] Connected to a master server at 127.0.99.60:35391
I20240503 07:59:36.633551 1833 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.633749 1833 heartbeater.cc:511] Master 127.0.99.60:35391 requested a full tablet report, sending...
I20240503 07:59:36.633769 1535 ts_manager.cc:194] Registered new tserver with Master: 10517edc7ed743fc888ce2aa19fbb6b3 (127.0.99.2:39847)
I20240503 07:59:36.633984 1832 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46757
I20240503 07:59:36.634100 1832 heartbeater.cc:464] Registering TS with master...
W20240503 07:59:36.634204 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:36.634299 1832 heartbeater.cc:511] Master 127.0.99.62:46757 requested a full tablet report, sending...
I20240503 07:59:36.634347 1601 ts_manager.cc:194] Registered new tserver with Master: 10517edc7ed743fc888ce2aa19fbb6b3 (127.0.99.2:39847)
I20240503 07:59:36.634440 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:36.634485 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:36.634513 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723176634512 us; error 0 us; skew 500 ppm
I20240503 07:59:36.634985 1469 ts_manager.cc:194] Registered new tserver with Master: 10517edc7ed743fc888ce2aa19fbb6b3 (127.0.99.2:39847)
I20240503 07:59:36.635635 396 webserver.cc:460] Webserver started at http://127.0.99.3:42723/ using document root <none> and password file <none>
I20240503 07:59:36.635794 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:36.635851 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:36.635927 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:36.636360 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "938eae137e584138a96d2405007d9e2b"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.637852 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.638861 1847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.639132 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.639206 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "938eae137e584138a96d2405007d9e2b"
format_stamp: "Formatted at 2024-05-03 07:59:36 on dist-test-slave-mxwh"
I20240503 07:59:36.639263 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:36.657609 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:36.658035 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:36.658318 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:36.658794 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:36.658836 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.658869 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:36.658900 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:36.663411 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:37481
I20240503 07:59:36.663487 1910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:37481 every 8 connection(s)
I20240503 07:59:36.668676 1913 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45211
I20240503 07:59:36.668799 1913 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.669029 1913 heartbeater.cc:511] Master 127.0.99.61:45211 requested a full tablet report, sending...
I20240503 07:59:36.669364 1911 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46757
I20240503 07:59:36.669447 1911 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.669646 1911 heartbeater.cc:511] Master 127.0.99.62:46757 requested a full tablet report, sending...
I20240503 07:59:36.669754 1535 ts_manager.cc:194] Registered new tserver with Master: 938eae137e584138a96d2405007d9e2b (127.0.99.3:37481)
I20240503 07:59:36.670030 1912 heartbeater.cc:347] Connected to a master server at 127.0.99.60:35391
I20240503 07:59:36.670101 1912 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:36.670248 1469 ts_manager.cc:194] Registered new tserver with Master: 938eae137e584138a96d2405007d9e2b (127.0.99.3:37481)
I20240503 07:59:36.670266 1912 heartbeater.cc:511] Master 127.0.99.60:35391 requested a full tablet report, sending...
I20240503 07:59:36.670812 1601 ts_manager.cc:194] Registered new tserver with Master: 938eae137e584138a96d2405007d9e2b (127.0.99.3:37481)
I20240503 07:59:36.671558 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007604604s
I20240503 07:59:36.703400 1644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.703583 1644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.704003 1644 leader_election.cc:288] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 752af6ee40b44ac29c3964194201da17 (127.0.99.62:46757), 88105ab869a6409e8428dc40caaf9b2a (127.0.99.61:45211)
I20240503 07:59:36.704528 1544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88105ab869a6409e8428dc40caaf9b2a" is_pre_election: true
I20240503 07:59:36.704528 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752af6ee40b44ac29c3964194201da17" is_pre_election: true
I20240503 07:59:36.704738 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcb9ed8c8564302bb686ae2bef06a47 in term 0.
I20240503 07:59:36.704738 1544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcb9ed8c8564302bb686ae2bef06a47 in term 0.
I20240503 07:59:36.705142 1585 leader_election.cc:302] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dcb9ed8c8564302bb686ae2bef06a47, 752af6ee40b44ac29c3964194201da17, 88105ab869a6409e8428dc40caaf9b2a; no voters:
I20240503 07:59:36.705335 1644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.705401 1644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.705430 1644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.706357 1644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.706701 1644 leader_election.cc:288] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [CANDIDATE]: Term 1 election: Requested vote from peers 752af6ee40b44ac29c3964194201da17 (127.0.99.62:46757), 88105ab869a6409e8428dc40caaf9b2a (127.0.99.61:45211)
I20240503 07:59:36.707007 1544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88105ab869a6409e8428dc40caaf9b2a"
I20240503 07:59:36.707006 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752af6ee40b44ac29c3964194201da17"
I20240503 07:59:36.707145 1544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.707160 1479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.708048 1544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcb9ed8c8564302bb686ae2bef06a47 in term 1.
I20240503 07:59:36.708043 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcb9ed8c8564302bb686ae2bef06a47 in term 1.
I20240503 07:59:36.708425 1585 leader_election.cc:302] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dcb9ed8c8564302bb686ae2bef06a47, 88105ab869a6409e8428dc40caaf9b2a; no voters:
I20240503 07:59:36.708603 1644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.708810 1644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 1 LEADER]: Becoming Leader. State: Replica: 2dcb9ed8c8564302bb686ae2bef06a47, State: Running, Role: LEADER
I20240503 07:59:36.708953 1644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:36.709635 1919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dcb9ed8c8564302bb686ae2bef06a47. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.709757 1919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:36.710067 1921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:36.710772 1921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:36.712568 1544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Refusing update from remote peer 2dcb9ed8c8564302bb686ae2bef06a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.712582 1479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Refusing update from remote peer 2dcb9ed8c8564302bb686ae2bef06a47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.712957 1919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.713101 1644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.714291 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dcb9ed8c8564302bb686ae2bef06a47. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.714313 1645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dcb9ed8c8564302bb686ae2bef06a47. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.714393 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.714414 1645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.715667 1919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.715778 1919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:36.715834 1928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.715934 1928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:36.716217 1921 catalog_manager.cc:1347] Generated new cluster ID: 048035619737437cbad91647cf9291d0
I20240503 07:59:36.716264 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.716413 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.716432 1921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:36.716615 1924 mvcc.cc:205] Tried to move back new op lower bound from 7023506131811119104 to 7023506131801366528. Current Snapshot: MvccSnapshot[applied={T|T < 7023506131811119104 or (T in {7023506131811119104})}]
I20240503 07:59:36.717190 1645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:36.717303 1645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: This master's current role is: FOLLOWER
I20240503 07:59:36.727375 1921 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:36.727994 1921 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:36.734781 1921 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: Generated new TSK 0
I20240503 07:59:36.734988 1921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:37.559209 1667 catalog_manager.cc:1259] Loaded cluster ID: 048035619737437cbad91647cf9291d0
I20240503 07:59:37.559309 1667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: loading cluster ID for follower catalog manager: success
I20240503 07:59:37.560660 1667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: acquiring CA information for follower catalog manager: success
I20240503 07:59:37.561182 1667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 07:59:37.565451 1680 catalog_manager.cc:1259] Loaded cluster ID: 048035619737437cbad91647cf9291d0
I20240503 07:59:37.565517 1680 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: loading cluster ID for follower catalog manager: success
I20240503 07:59:37.566588 1680 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: acquiring CA information for follower catalog manager: success
I20240503 07:59:37.567042 1680 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 07:59:37.603662 1601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42296
I20240503 07:59:37.635978 1601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42304
I20240503 07:59:37.672273 1601 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42312
I20240503 07:59:38.557044 1660 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:38.604583 1754 heartbeater.cc:503] Master 127.0.99.60:35391 was elected leader, sending a full tablet report...
I20240503 07:59:38.636936 1833 heartbeater.cc:503] Master 127.0.99.60:35391 was elected leader, sending a full tablet report...
I20240503 07:59:38.673060 1912 heartbeater.cc:503] Master 127.0.99.60:35391 was elected leader, sending a full tablet report...
I20240503 07:59:38.835594 396 test_util.cc:268] Using random seed: 435580550
I20240503 07:59:38.844516 1601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42320:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 07:59:38.859122 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 5029f6df11c24b29bfbafb24e0778682 (DEFAULT_TABLE table=test-workload [id=14f6120449904052af7b25e511bc9db6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 07:59:38.859442 1717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5029f6df11c24b29bfbafb24e0778682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.860635 1875 tablet_service.cc:1467] Processing CreateTablet for tablet 77968242acde4a8d977f88d122efae4d (DEFAULT_TABLE table=test-workload [id=14f6120449904052af7b25e511bc9db6]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 07:59:38.860710 1796 tablet_service.cc:1467] Processing CreateTablet for tablet 4c6b6aec2330499e9c0be2b1c21db031 (DEFAULT_TABLE table=test-workload [id=14f6120449904052af7b25e511bc9db6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 07:59:38.860922 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c6b6aec2330499e9c0be2b1c21db031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.860922 1875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77968242acde4a8d977f88d122efae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.861783 1960 tablet_bootstrap.cc:492] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: Bootstrap starting.
I20240503 07:59:38.862639 1960 tablet_bootstrap.cc:654] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.863081 1962 tablet_bootstrap.cc:492] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: Bootstrap starting.
I20240503 07:59:38.863296 1961 tablet_bootstrap.cc:492] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: Bootstrap starting.
I20240503 07:59:38.863819 1960 tablet_bootstrap.cc:492] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: No bootstrap required, opened a new log
I20240503 07:59:38.863905 1960 ts_tablet_manager.cc:1367] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.864236 1962 tablet_bootstrap.cc:654] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.864373 1960 raft_consensus.cc:348] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 40255 } }
I20240503 07:59:38.864490 1960 raft_consensus.cc:374] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.864544 1960 raft_consensus.cc:729] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af22abc6a9a24fa4b45ff5e4a41098ca, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.864682 1960 consensus_queue.cc:260] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 40255 } }
I20240503 07:59:38.864763 1960 raft_consensus.cc:388] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:38.864810 1960 raft_consensus.cc:482] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:38.864879 1960 raft_consensus.cc:3037] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.864991 1961 tablet_bootstrap.cc:654] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.865497 1962 tablet_bootstrap.cc:492] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: No bootstrap required, opened a new log
I20240503 07:59:38.865607 1962 ts_tablet_manager.cc:1367] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.865618 1960 raft_consensus.cc:504] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 40255 } }
I20240503 07:59:38.865820 1960 leader_election.cc:302] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af22abc6a9a24fa4b45ff5e4a41098ca; no voters:
I20240503 07:59:38.866024 1960 leader_election.cc:288] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:38.866114 1962 raft_consensus.cc:348] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938eae137e584138a96d2405007d9e2b" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37481 } }
I20240503 07:59:38.866185 1966 raft_consensus.cc:2781] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.866286 1962 raft_consensus.cc:374] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.866323 1962 raft_consensus.cc:729] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938eae137e584138a96d2405007d9e2b, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.866366 1960 ts_tablet_manager.cc:1398] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:38.866437 1966 raft_consensus.cc:686] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 1 LEADER]: Becoming Leader. State: Replica: af22abc6a9a24fa4b45ff5e4a41098ca, State: Running, Role: LEADER
I20240503 07:59:38.866564 1966 consensus_queue.cc:237] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 40255 } }
I20240503 07:59:38.866672 1962 consensus_queue.cc:260] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938eae137e584138a96d2405007d9e2b" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37481 } }
I20240503 07:59:38.866742 1962 raft_consensus.cc:388] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:38.866772 1962 raft_consensus.cc:482] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:38.866814 1962 raft_consensus.cc:3037] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.866214 1961 tablet_bootstrap.cc:492] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: No bootstrap required, opened a new log
I20240503 07:59:38.867076 1961 ts_tablet_manager.cc:1367] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:38.867502 1961 raft_consensus.cc:348] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39847 } }
I20240503 07:59:38.867615 1961 raft_consensus.cc:374] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.867666 1961 raft_consensus.cc:729] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10517edc7ed743fc888ce2aa19fbb6b3, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.867810 1961 consensus_queue.cc:260] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39847 } }
I20240503 07:59:38.867933 1961 raft_consensus.cc:388] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:38.867982 1961 raft_consensus.cc:482] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:38.868031 1961 raft_consensus.cc:3037] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.868791 1962 raft_consensus.cc:504] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938eae137e584138a96d2405007d9e2b" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37481 } }
I20240503 07:59:38.868965 1962 leader_election.cc:302] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938eae137e584138a96d2405007d9e2b; no voters:
I20240503 07:59:38.869004 1601 catalog_manager.cc:5484] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca reported cstate change: term changed from 0 to 1, leader changed from <none> to af22abc6a9a24fa4b45ff5e4a41098ca (127.0.99.1). New cstate: current_term: 1 leader_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af22abc6a9a24fa4b45ff5e4a41098ca" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 40255 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.869156 1962 leader_election.cc:288] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:38.869407 1962 ts_tablet_manager.cc:1398] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:38.869680 1961 raft_consensus.cc:504] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39847 } }
I20240503 07:59:38.869822 1961 leader_election.cc:302] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10517edc7ed743fc888ce2aa19fbb6b3; no voters:
I20240503 07:59:38.870024 1961 leader_election.cc:288] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:38.870306 1974 raft_consensus.cc:2781] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.870368 1961 ts_tablet_manager.cc:1398] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 07:59:38.870499 1974 raft_consensus.cc:686] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 10517edc7ed743fc888ce2aa19fbb6b3, State: Running, Role: LEADER
I20240503 07:59:38.870965 1971 raft_consensus.cc:2781] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.871467 1971 raft_consensus.cc:686] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 1 LEADER]: Becoming Leader. State: Replica: 938eae137e584138a96d2405007d9e2b, State: Running, Role: LEADER
I20240503 07:59:38.871564 1971 consensus_queue.cc:237] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938eae137e584138a96d2405007d9e2b" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37481 } }
I20240503 07:59:38.870658 1974 consensus_queue.cc:237] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39847 } }
I20240503 07:59:38.876327 1599 catalog_manager.cc:5484] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 938eae137e584138a96d2405007d9e2b (127.0.99.3). New cstate: current_term: 1 leader_uuid: "938eae137e584138a96d2405007d9e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938eae137e584138a96d2405007d9e2b" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37481 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.876925 1600 catalog_manager.cc:5484] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10517edc7ed743fc888ce2aa19fbb6b3 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10517edc7ed743fc888ce2aa19fbb6b3" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39847 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.882412 396 master.cc:537] Master@127.0.99.60:35391 shutting down...
I20240503 07:59:38.885588 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:38.885782 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.885847 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2dcb9ed8c8564302bb686ae2bef06a47: stopping tablet replica
I20240503 07:59:38.899843 396 master.cc:552] Master@127.0.99.60:35391 shutdown complete.
W20240503 07:59:39.878508 1754 heartbeater.cc:644] Failed to heartbeat to 127.0.99.60:35391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.99.60:35391: connect: Connection refused (error 111)
I20240503 07:59:40.381467 1983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2dcb9ed8c8564302bb686ae2bef06a47)
I20240503 07:59:40.381623 1983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:40.382091 1983 leader_election.cc:288] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 752af6ee40b44ac29c3964194201da17 (127.0.99.62:46757), 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391)
I20240503 07:59:40.382555 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88105ab869a6409e8428dc40caaf9b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "752af6ee40b44ac29c3964194201da17" is_pre_election: true
I20240503 07:59:40.382624 1984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2dcb9ed8c8564302bb686ae2bef06a47)
I20240503 07:59:40.382726 1984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:40.382953 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88105ab869a6409e8428dc40caaf9b2a in term 1.
I20240503 07:59:40.383263 1521 leader_election.cc:302] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752af6ee40b44ac29c3964194201da17, 88105ab869a6409e8428dc40caaf9b2a; no voters:
I20240503 07:59:40.383412 1984 leader_election.cc:288] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88105ab869a6409e8428dc40caaf9b2a (127.0.99.61:45211), 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391)
I20240503 07:59:40.383419 1983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:40.383796 1983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2dcb9ed8c8564302bb686ae2bef06a47)
I20240503 07:59:40.383427 1544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752af6ee40b44ac29c3964194201da17" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "88105ab869a6409e8428dc40caaf9b2a" is_pre_election: true
I20240503 07:59:40.383849 1983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:40.384001 1522 leader_election.cc:334] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391): Network error: Client connection negotiation failed: client connection to 127.0.99.60:35391: connect: Connection refused (error 111)
W20240503 07:59:40.385154 1456 leader_election.cc:334] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391): Network error: Client connection negotiation failed: client connection to 127.0.99.60:35391: connect: Connection refused (error 111)
I20240503 07:59:40.386327 1983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:40.386567 1544 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 752af6ee40b44ac29c3964194201da17 in current term 2: Already voted for candidate 88105ab869a6409e8428dc40caaf9b2a in this term.
I20240503 07:59:40.386772 1983 leader_election.cc:288] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 election: Requested vote from peers 752af6ee40b44ac29c3964194201da17 (127.0.99.62:46757), 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391)
I20240503 07:59:40.387157 1453 leader_election.cc:302] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752af6ee40b44ac29c3964194201da17; no voters: 2dcb9ed8c8564302bb686ae2bef06a47, 88105ab869a6409e8428dc40caaf9b2a
I20240503 07:59:40.387156 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88105ab869a6409e8428dc40caaf9b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "752af6ee40b44ac29c3964194201da17"
I20240503 07:59:40.387491 1984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:40.387894 1522 leader_election.cc:334] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2dcb9ed8c8564302bb686ae2bef06a47 (127.0.99.60:35391): Network error: Client connection negotiation failed: client connection to 127.0.99.60:35391: connect: Connection refused (error 111)
I20240503 07:59:40.388463 1984 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:40.389397 1479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88105ab869a6409e8428dc40caaf9b2a in term 2.
I20240503 07:59:40.389842 1521 leader_election.cc:302] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 752af6ee40b44ac29c3964194201da17, 88105ab869a6409e8428dc40caaf9b2a; no voters: 2dcb9ed8c8564302bb686ae2bef06a47
I20240503 07:59:40.390030 1983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:40.390230 1983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 LEADER]: Becoming Leader. State: Replica: 88105ab869a6409e8428dc40caaf9b2a, State: Running, Role: LEADER
I20240503 07:59:40.390374 1983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } }
I20240503 07:59:40.391024 1989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88105ab869a6409e8428dc40caaf9b2a. Latest consensus state: current_term: 2 leader_uuid: "88105ab869a6409e8428dc40caaf9b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752af6ee40b44ac29c3964194201da17" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 46757 } } peers { permanent_uuid: "88105ab869a6409e8428dc40caaf9b2a" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45211 } } peers { permanent_uuid: "2dcb9ed8c8564302bb686ae2bef06a47" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 35391 } } }
I20240503 07:59:40.391150 1989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:40.391407 1991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:40.392469 1991 catalog_manager.cc:669] Loaded metadata for table test-workload [id=14f6120449904052af7b25e511bc9db6]
I20240503 07:59:40.393169 1991 tablet_loader.cc:96] loaded metadata for tablet 4c6b6aec2330499e9c0be2b1c21db031 (table test-workload [id=14f6120449904052af7b25e511bc9db6])
I20240503 07:59:40.393370 1991 tablet_loader.cc:96] loaded metadata for tablet 5029f6df11c24b29bfbafb24e0778682 (table test-workload [id=14f6120449904052af7b25e511bc9db6])
I20240503 07:59:40.393537 1991 tablet_loader.cc:96] loaded metadata for tablet 77968242acde4a8d977f88d122efae4d (table test-workload [id=14f6120449904052af7b25e511bc9db6])
I20240503 07:59:40.393625 1991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:40.393990 1991 catalog_manager.cc:1259] Loaded cluster ID: 048035619737437cbad91647cf9291d0
I20240503 07:59:40.394033 1991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:40.394543 1991 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:40.394943 1991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: Loaded TSK: 0
I20240503 07:59:40.395387 1991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:40.559237 1666 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:40.559335 1666 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:40.666190 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 07:59:40.670271 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:40.670477 396 tablet_replica.cc:330] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca: stopping tablet replica
I20240503 07:59:40.670580 396 raft_consensus.cc:2229] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:40.670704 396 raft_consensus.cc:2258] T 5029f6df11c24b29bfbafb24e0778682 P af22abc6a9a24fa4b45ff5e4a41098ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.683137 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 07:59:40.685045 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 07:59:40.689189 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:40.689364 396 tablet_replica.cc:330] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3: stopping tablet replica
I20240503 07:59:40.689463 396 raft_consensus.cc:2229] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:40.689571 396 raft_consensus.cc:2258] T 4c6b6aec2330499e9c0be2b1c21db031 P 10517edc7ed743fc888ce2aa19fbb6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.702049 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 07:59:40.703927 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 07:59:40.708026 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 07:59:40.708221 396 tablet_replica.cc:330] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b: stopping tablet replica
I20240503 07:59:40.708334 396 raft_consensus.cc:2229] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:40.708451 396 raft_consensus.cc:2258] T 77968242acde4a8d977f88d122efae4d P 938eae137e584138a96d2405007d9e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.721755 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 07:59:40.723680 396 master.cc:537] Master@127.0.99.62:46757 shutting down...
I20240503 07:59:40.727397 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.727546 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.727612 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 752af6ee40b44ac29c3964194201da17: stopping tablet replica
I20240503 07:59:40.740056 396 master.cc:552] Master@127.0.99.62:46757 shutdown complete.
I20240503 07:59:40.742715 396 master.cc:537] Master@127.0.99.61:45211 shutting down...
I20240503 07:59:40.746408 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 LEADER]: Raft consensus shutting down.
I20240503 07:59:40.746598 396 pending_rounds.cc:62] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: Trying to abort 1 pending ops.
I20240503 07:59:40.746654 396 pending_rounds.cc:68] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7023506146880413696 op_type: NO_OP noop_request { }
I20240503 07:59:40.746737 396 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 07:59:40.746788 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.746884 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88105ab869a6409e8428dc40caaf9b2a: stopping tablet replica
I20240503 07:59:40.749215 396 master.cc:552] Master@127.0.99.61:45211 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4311 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 07:59:40.756115 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:44675
I20240503 07:59:40.756440 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.759009 1993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.759969 1995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.760881 1992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.761695 396 server_base.cc:1034] running on GCE node
I20240503 07:59:40.761864 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:40.761902 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:40.761926 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723180761925 us; error 0 us; skew 500 ppm
I20240503 07:59:40.768131 396 webserver.cc:460] Webserver started at http://127.0.99.62:36965/ using document root <none> and password file <none>
I20240503 07:59:40.768342 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.768396 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.768476 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.768870 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "44bfdf1b07304b30920566b08f3bf8c1"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.770311 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:40.771234 2000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.771548 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.771617 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "44bfdf1b07304b30920566b08f3bf8c1"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.771705 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.784088 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.784446 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.788312 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:44675
I20240503 07:59:40.791219 2052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:44675 every 8 connection(s)
I20240503 07:59:40.791646 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.793363 2053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1: Bootstrap starting.
I20240503 07:59:40.794040 2053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.794939 2053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1: No bootstrap required, opened a new log
I20240503 07:59:40.795264 2053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER }
I20240503 07:59:40.795363 2053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.795397 2053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bfdf1b07304b30920566b08f3bf8c1, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.795528 2053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER }
I20240503 07:59:40.795598 2053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:40.795636 2053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:40.795686 2053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.796337 2053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER }
I20240503 07:59:40.796445 2053 leader_election.cc:302] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44bfdf1b07304b30920566b08f3bf8c1; no voters:
I20240503 07:59:40.796599 2053 leader_election.cc:288] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:40.796690 2056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.796876 2056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 1 LEADER]: Becoming Leader. State: Replica: 44bfdf1b07304b30920566b08f3bf8c1, State: Running, Role: LEADER
I20240503 07:59:40.796991 2053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:40.797005 2056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER }
I20240503 07:59:40.797431 2057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44bfdf1b07304b30920566b08f3bf8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER } }
I20240503 07:59:40.797472 2058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44bfdf1b07304b30920566b08f3bf8c1. Latest consensus state: current_term: 1 leader_uuid: "44bfdf1b07304b30920566b08f3bf8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bfdf1b07304b30920566b08f3bf8c1" member_type: VOTER } }
I20240503 07:59:40.797581 2057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:40.797672 2058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:40.798017 2064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:40.798754 2064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:40.799013 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 07:59:40.800465 2064 catalog_manager.cc:1347] Generated new cluster ID: 8d71b1ac415440da83997c791cfacfc8
I20240503 07:59:40.800518 2064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:40.810379 2064 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:40.810891 2064 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:40.820822 2064 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1: Generated new TSK 0
I20240503 07:59:40.820984 2064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:40.831300 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.834120 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.834986 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.835873 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.836652 396 server_base.cc:1034] running on GCE node
I20240503 07:59:40.836823 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:40.836863 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:40.836889 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723180836887 us; error 0 us; skew 500 ppm
I20240503 07:59:40.837792 396 webserver.cc:460] Webserver started at http://127.0.99.1:40889/ using document root <none> and password file <none>
I20240503 07:59:40.837932 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.837985 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.838063 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.838444 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "6e48e9f07afa4101bd939f37a16cfbdb"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.839823 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:40.840708 2082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.840966 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:40.841027 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "6e48e9f07afa4101bd939f37a16cfbdb"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.841081 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.884737 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.885093 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.885344 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.885802 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.885844 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.885876 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.885905 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.890220 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:44483
I20240503 07:59:40.890265 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:44483 every 8 connection(s)
I20240503 07:59:40.890985 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.893839 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.894762 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.895202 2146 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44675
I20240503 07:59:40.895287 2146 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.895467 2146 heartbeater.cc:511] Master 127.0.99.62:44675 requested a full tablet report, sending...
W20240503 07:59:40.895663 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.896081 2017 ts_manager.cc:194] Registered new tserver with Master: 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483)
I20240503 07:59:40.896428 396 server_base.cc:1034] running on GCE node
I20240503 07:59:40.896585 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:40.896636 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:40.896657 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723180896657 us; error 0 us; skew 500 ppm
I20240503 07:59:40.897123 2017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51094
I20240503 07:59:40.897688 396 webserver.cc:460] Webserver started at http://127.0.99.2:33023/ using document root <none> and password file <none>
I20240503 07:59:40.897840 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.897898 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.897969 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.898378 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "66527bcbf3a34a158e1595e44b4ea248"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.899760 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.900684 2158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.900902 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:40.900961 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "66527bcbf3a34a158e1595e44b4ea248"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.901026 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.906167 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.906466 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.906704 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.907156 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.907196 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.907244 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.907272 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.911034 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:34119
I20240503 07:59:40.911089 2221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:34119 every 8 connection(s)
I20240503 07:59:40.911758 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.914696 2225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.915424 2222 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44675
I20240503 07:59:40.915508 2222 heartbeater.cc:464] Registering TS with master...
W20240503 07:59:40.915658 2226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.915709 2222 heartbeater.cc:511] Master 127.0.99.62:44675 requested a full tablet report, sending...
I20240503 07:59:40.916297 2017 ts_manager.cc:194] Registered new tserver with Master: 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2:34119)
I20240503 07:59:40.916640 396 server_base.cc:1034] running on GCE node
I20240503 07:59:40.916922 2017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51102
W20240503 07:59:40.917460 2228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.917670 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:40.917716 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:40.917732 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723180917731 us; error 0 us; skew 500 ppm
I20240503 07:59:40.918632 396 webserver.cc:460] Webserver started at http://127.0.99.3:46103/ using document root <none> and password file <none>
I20240503 07:59:40.918766 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.918812 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.918895 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.919291 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.920733 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:40.921548 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.921749 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.921813 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-mxwh"
I20240503 07:59:40.921882 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.941681 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.942081 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.942391 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.942838 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.942878 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.942929 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.942956 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.947185 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:34233
I20240503 07:59:40.947238 2296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:34233 every 8 connection(s)
I20240503 07:59:40.956714 2297 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44675
I20240503 07:59:40.956838 2297 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.957084 2297 heartbeater.cc:511] Master 127.0.99.62:44675 requested a full tablet report, sending...
I20240503 07:59:40.958029 2017 ts_manager.cc:194] Registered new tserver with Master: 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3:34233)
I20240503 07:59:40.958779 2017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51116
I20240503 07:59:40.958863 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011290947s
I20240503 07:59:41.898133 2146 heartbeater.cc:503] Master 127.0.99.62:44675 was elected leader, sending a full tablet report...
I20240503 07:59:41.917811 2222 heartbeater.cc:503] Master 127.0.99.62:44675 was elected leader, sending a full tablet report...
I20240503 07:59:41.959887 2297 heartbeater.cc:503] Master 127.0.99.62:44675 was elected leader, sending a full tablet report...
I20240503 07:59:41.984947 396 test_util.cc:268] Using random seed: 438729904
I20240503 07:59:41.993566 2017 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51118:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 07:59:41.994091 2017 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 07:59:42.003279 2186 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7a1a459ba4433da09a0a3e7efac6ea (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 07:59:42.003624 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7a1a459ba4433da09a0a3e7efac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.003930 2185 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b086cb6a9411eac66da6d2506e590 (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 07:59:42.004225 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b086cb6a9411eac66da6d2506e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.005221 2110 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7a1a459ba4433da09a0a3e7efac6ea (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 07:59:42.005467 2110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7a1a459ba4433da09a0a3e7efac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.006454 2109 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b086cb6a9411eac66da6d2506e590 (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 07:59:42.006650 2109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b086cb6a9411eac66da6d2506e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.007016 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7a1a459ba4433da09a0a3e7efac6ea (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 07:59:42.007297 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7a1a459ba4433da09a0a3e7efac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.008289 2318 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: Bootstrap starting.
I20240503 07:59:42.008478 2317 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: Bootstrap starting.
I20240503 07:59:42.009356 2317 tablet_bootstrap.cc:654] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.009380 2318 tablet_bootstrap.cc:654] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.009665 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b086cb6a9411eac66da6d2506e590 (DEFAULT_TABLE table=test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 07:59:42.009935 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b086cb6a9411eac66da6d2506e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:42.010241 2319 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: Bootstrap starting.
I20240503 07:59:42.010792 2318 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: No bootstrap required, opened a new log
I20240503 07:59:42.010936 2318 ts_tablet_manager.cc:1367] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:42.011073 2319 tablet_bootstrap.cc:654] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.011505 2318 raft_consensus.cc:348] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.011700 2318 raft_consensus.cc:374] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.011790 2318 raft_consensus.cc:729] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e48e9f07afa4101bd939f37a16cfbdb, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.011996 2318 consensus_queue.cc:260] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.012400 2317 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: No bootstrap required, opened a new log
I20240503 07:59:42.012467 2318 ts_tablet_manager.cc:1398] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:42.012493 2317 ts_tablet_manager.cc:1367] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:42.012526 2319 tablet_bootstrap.cc:492] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: No bootstrap required, opened a new log
I20240503 07:59:42.012583 2319 ts_tablet_manager.cc:1367] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:42.012660 2318 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Bootstrap starting.
I20240503 07:59:42.012955 2319 raft_consensus.cc:348] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.013033 2317 raft_consensus.cc:348] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.013059 2319 raft_consensus.cc:374] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.013118 2317 raft_consensus.cc:374] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.013154 2319 raft_consensus.cc:729] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a25c2ccb594a4da9ce40ec51e3b149, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.013154 2317 raft_consensus.cc:729] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66527bcbf3a34a158e1595e44b4ea248, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.013324 2319 consensus_queue.cc:260] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.013355 2317 consensus_queue.cc:260] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.013684 2319 ts_tablet_manager.cc:1398] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:42.013684 2317 ts_tablet_manager.cc:1398] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:42.013962 2319 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: Bootstrap starting.
I20240503 07:59:42.013981 2317 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: Bootstrap starting.
I20240503 07:59:42.014775 2317 tablet_bootstrap.cc:654] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.013671 2318 tablet_bootstrap.cc:654] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.014914 2319 tablet_bootstrap.cc:654] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:42.017033 2317 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: No bootstrap required, opened a new log
I20240503 07:59:42.017081 2318 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: No bootstrap required, opened a new log
I20240503 07:59:42.017140 2317 ts_tablet_manager.cc:1367] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:42.017172 2318 ts_tablet_manager.cc:1367] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:42.017580 2317 raft_consensus.cc:348] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.017706 2317 raft_consensus.cc:374] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.017592 2318 raft_consensus.cc:348] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.017742 2317 raft_consensus.cc:729] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66527bcbf3a34a158e1595e44b4ea248, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.017764 2318 raft_consensus.cc:374] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.017798 2318 raft_consensus.cc:729] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e48e9f07afa4101bd939f37a16cfbdb, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.017863 2317 consensus_queue.cc:260] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.017966 2319 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: No bootstrap required, opened a new log
I20240503 07:59:42.017916 2318 consensus_queue.cc:260] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.018052 2319 ts_tablet_manager.cc:1367] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:42.018128 2317 ts_tablet_manager.cc:1398] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:42.018222 2318 ts_tablet_manager.cc:1398] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:42.018501 2319 raft_consensus.cc:348] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.018646 2319 raft_consensus.cc:374] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:42.018699 2319 raft_consensus.cc:729] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a25c2ccb594a4da9ce40ec51e3b149, State: Initialized, Role: FOLLOWER
I20240503 07:59:42.018850 2319 consensus_queue.cc:260] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.019203 2319 ts_tablet_manager.cc:1398] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:42.076114 2325 raft_consensus.cc:482] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:42.076341 2325 raft_consensus.cc:504] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.076869 2325 leader_election.cc:288] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3:34233), 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483)
I20240503 07:59:42.080381 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" is_pre_election: true
I20240503 07:59:42.080448 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" is_pre_election: true
I20240503 07:59:42.080523 2120 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 0.
I20240503 07:59:42.080575 2271 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 0.
I20240503 07:59:42.080821 2159 leader_election.cc:302] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66527bcbf3a34a158e1595e44b4ea248, 6e48e9f07afa4101bd939f37a16cfbdb; no voters:
I20240503 07:59:42.080961 2325 raft_consensus.cc:2781] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:42.081017 2325 raft_consensus.cc:482] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:42.081043 2325 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.081876 2325 raft_consensus.cc:504] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.082214 2325 leader_election.cc:288] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 election: Requested vote from peers 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3:34233), 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483)
I20240503 07:59:42.082507 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
I20240503 07:59:42.082583 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e48e9f07afa4101bd939f37a16cfbdb"
I20240503 07:59:42.082608 2271 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.082690 2120 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.083441 2271 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 1.
I20240503 07:59:42.083541 2120 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 1.
I20240503 07:59:42.083812 2160 leader_election.cc:302] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a25c2ccb594a4da9ce40ec51e3b149, 66527bcbf3a34a158e1595e44b4ea248; no voters:
I20240503 07:59:42.084033 2325 raft_consensus.cc:2781] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:42.084354 2325 raft_consensus.cc:686] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Becoming Leader. State: Replica: 66527bcbf3a34a158e1595e44b4ea248, State: Running, Role: LEADER
I20240503 07:59:42.084568 2325 consensus_queue.cc:237] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:42.086201 2016 catalog_manager.cc:5484] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "66527bcbf3a34a158e1595e44b4ea248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:42.350033 2325 raft_consensus.cc:482] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:42.350210 2325 raft_consensus.cc:504] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.350641 2325 leader_election.cc:288] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483), 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3:34233)
I20240503 07:59:42.351161 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b086cb6a9411eac66da6d2506e590" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" is_pre_election: true
I20240503 07:59:42.351161 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b086cb6a9411eac66da6d2506e590" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" is_pre_election: true
I20240503 07:59:42.351343 2271 raft_consensus.cc:2454] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 0.
I20240503 07:59:42.351341 2120 raft_consensus.cc:2454] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 0.
I20240503 07:59:42.351652 2159 leader_election.cc:302] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66527bcbf3a34a158e1595e44b4ea248, 6e48e9f07afa4101bd939f37a16cfbdb; no voters:
I20240503 07:59:42.351840 2325 raft_consensus.cc:2781] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:42.351907 2325 raft_consensus.cc:482] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:42.351943 2325 raft_consensus.cc:3037] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.352797 2325 raft_consensus.cc:504] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.353114 2325 leader_election.cc:288] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 election: Requested vote from peers 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483), 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3:34233)
I20240503 07:59:42.353504 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b086cb6a9411eac66da6d2506e590" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e48e9f07afa4101bd939f37a16cfbdb"
I20240503 07:59:42.353504 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b086cb6a9411eac66da6d2506e590" candidate_uuid: "66527bcbf3a34a158e1595e44b4ea248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
I20240503 07:59:42.353668 2271 raft_consensus.cc:3037] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.353683 2120 raft_consensus.cc:3037] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:42.354602 2271 raft_consensus.cc:2454] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 1.
I20240503 07:59:42.354602 2120 raft_consensus.cc:2454] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66527bcbf3a34a158e1595e44b4ea248 in term 1.
I20240503 07:59:42.354904 2160 leader_election.cc:302] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a25c2ccb594a4da9ce40ec51e3b149, 66527bcbf3a34a158e1595e44b4ea248; no voters:
I20240503 07:59:42.355113 2325 raft_consensus.cc:2781] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:42.355183 2325 raft_consensus.cc:686] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Becoming Leader. State: Replica: 66527bcbf3a34a158e1595e44b4ea248, State: Running, Role: LEADER
I20240503 07:59:42.355324 2325 consensus_queue.cc:237] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } }
I20240503 07:59:42.356894 2016 catalog_manager.cc:5484] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "66527bcbf3a34a158e1595e44b4ea248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:42.479151 2325 consensus_queue.cc:1035] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:42.482384 2331 consensus_queue.cc:1035] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:42.791889 2331 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:42.794875 2337 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:42.798748 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:42.803339 2196 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea"
dest_uuid: "66527bcbf3a34a158e1595e44b4ea248"
mode: GRACEFUL
new_leader_uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
from {username='slave'} at 127.0.0.1:41082
I20240503 07:59:42.803503 2196 raft_consensus.cc:595] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Received request to transfer leadership to TS 30a25c2ccb594a4da9ce40ec51e3b149
I20240503 07:59:42.803877 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 07:59:42.803985 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:43.011720 2337 raft_consensus.cc:979] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: : Instructing follower 30a25c2ccb594a4da9ce40ec51e3b149 to start an election
I20240503 07:59:43.011827 2337 raft_consensus.cc:1067] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Signalling peer 30a25c2ccb594a4da9ce40ec51e3b149 to start an election
I20240503 07:59:43.012439 2271 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea"
dest_uuid: "30a25c2ccb594a4da9ce40ec51e3b149"
from {username='slave'} at 127.0.0.1:49976
I20240503 07:59:43.012638 2271 raft_consensus.cc:482] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:43.012710 2271 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:43.013556 2271 raft_consensus.cc:504] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:43.013998 2271 leader_election.cc:288] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [CANDIDATE]: Term 2 election: Requested vote from peers 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2:34119), 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483)
I20240503 07:59:43.018616 2196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "66527bcbf3a34a158e1595e44b4ea248"
I20240503 07:59:43.018616 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7a1a459ba4433da09a0a3e7efac6ea" candidate_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e48e9f07afa4101bd939f37a16cfbdb"
I20240503 07:59:43.018815 2120 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:43.018815 2196 raft_consensus.cc:3032] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:43.018983 2196 raft_consensus.cc:729] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66527bcbf3a34a158e1595e44b4ea248, State: Running, Role: LEADER
I20240503 07:59:43.019182 2196 consensus_queue.cc:260] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:43.019384 2196 raft_consensus.cc:3037] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:43.019994 2120 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a25c2ccb594a4da9ce40ec51e3b149 in term 2.
I20240503 07:59:43.020360 2234 leader_election.cc:302] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a25c2ccb594a4da9ce40ec51e3b149, 6e48e9f07afa4101bd939f37a16cfbdb; no voters:
I20240503 07:59:43.020408 2196 raft_consensus.cc:2454] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a25c2ccb594a4da9ce40ec51e3b149 in term 2.
I20240503 07:59:43.020599 2324 raft_consensus.cc:2781] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:43.020844 2324 raft_consensus.cc:686] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 2 LEADER]: Becoming Leader. State: Replica: 30a25c2ccb594a4da9ce40ec51e3b149, State: Running, Role: LEADER
I20240503 07:59:43.020994 2324 consensus_queue.cc:237] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } }
I20240503 07:59:43.022440 2015 catalog_manager.cc:5484] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 reported cstate change: term changed from 1 to 2, leader changed from 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2) to 30a25c2ccb594a4da9ce40ec51e3b149 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:43.454880 2120 raft_consensus.cc:1261] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 2 FOLLOWER]: Refusing update from remote peer 30a25c2ccb594a4da9ce40ec51e3b149: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:43.455328 2324 consensus_queue.cc:1035] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:43.457971 2196 raft_consensus.cc:1261] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 2 FOLLOWER]: Refusing update from remote peer 30a25c2ccb594a4da9ce40ec51e3b149: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:43.458426 2353 consensus_queue.cc:1035] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:44.804347 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:44.804780 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:44.804869 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:46.805100 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:46.805475 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:46.805544 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:48.805817 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:48.806208 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:48.806281 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:50.806557 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:50.806953 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:50.807019 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:52.807466 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:52.808109 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:52.808220 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:53.426201 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.429474 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.430511 396 server_base.cc:1034] running on GCE node
W20240503 07:59:53.431344 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.432117 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.432421 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 07:59:53.432473 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 07:59:53.432492 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723193432491 us; error 0 us; skew 500 ppm
I20240503 07:59:53.433531 396 webserver.cc:460] Webserver started at http://127.0.99.4:37265/ using document root <none> and password file <none>
I20240503 07:59:53.433663 396 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.433707 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.433761 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:53.434130 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-3-root/instance:
uuid: "fe8c6b1251e54f1e87923acb428eb2c8"
format_stamp: "Formatted at 2024-05-03 07:59:53 on dist-test-slave-mxwh"
I20240503 07:59:53.435638 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:53.436645 2367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.436944 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:53.437009 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-3-root
uuid: "fe8c6b1251e54f1e87923acb428eb2c8"
format_stamp: "Formatted at 2024-05-03 07:59:53 on dist-test-slave-mxwh"
I20240503 07:59:53.437070 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.445900 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.446254 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.446518 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.446975 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:53.447016 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.447047 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:53.447063 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.451828 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:39737
I20240503 07:59:53.451866 2430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:39737 every 8 connection(s)
I20240503 07:59:53.456508 2431 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44675
I20240503 07:59:53.456619 2431 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.456828 2431 heartbeater.cc:511] Master 127.0.99.62:44675 requested a full tablet report, sending...
I20240503 07:59:53.457504 2015 ts_manager.cc:194] Registered new tserver with Master: fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4:39737)
I20240503 07:59:53.458046 2015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37828
I20240503 07:59:53.814720 2196 consensus_queue.cc:237] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } }
I20240503 07:59:53.816291 2120 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.816650 2271 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.816699 2356 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:53.816963 2440 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:53.818372 2356 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } } }
I20240503 07:59:53.818755 2120 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } } }
I20240503 07:59:53.819629 2271 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } } }
W20240503 07:59:53.820961 2162 consensus_peers.cc:485] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 -> Peer fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4:39737): Couldn't send request to peer fe8c6b1251e54f1e87923acb428eb2c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f4b086cb6a9411eac66da6d2506e590. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:53.821450 2015 catalog_manager.cc:5484] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 reported cstate change: config changed from index -1 to 2, NON_VOTER fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "66527bcbf3a34a158e1595e44b4ea248" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:54.388790 2450 ts_tablet_manager.cc:902] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Initiating tablet copy from peer 66527bcbf3a34a158e1595e44b4ea248 (127.0.99.2:34119)
I20240503 07:59:54.389611 2450 tablet_copy_client.cc:308] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34119
I20240503 07:59:54.393853 2206 tablet_copy_service.cc:140] P 66527bcbf3a34a158e1595e44b4ea248: Received BeginTabletCopySession request for tablet 1f4b086cb6a9411eac66da6d2506e590 from peer fe8c6b1251e54f1e87923acb428eb2c8 ({username='slave'} at 127.0.0.1:32828)
I20240503 07:59:54.393996 2206 tablet_copy_service.cc:161] P 66527bcbf3a34a158e1595e44b4ea248: Beginning new tablet copy session on tablet 1f4b086cb6a9411eac66da6d2506e590 from peer fe8c6b1251e54f1e87923acb428eb2c8 at {username='slave'} at 127.0.0.1:32828: session id = fe8c6b1251e54f1e87923acb428eb2c8-1f4b086cb6a9411eac66da6d2506e590
I20240503 07:59:54.395496 2206 tablet_copy_source_session.cc:205] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:54.396787 2450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b086cb6a9411eac66da6d2506e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:54.398739 2450 tablet_copy_client.cc:783] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:54.398887 2450 tablet_copy_client.cc:647] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:54.400501 2450 tablet_copy_client.cc:523] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:54.402145 2450 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Bootstrap starting.
I20240503 07:59:54.406456 2450 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:54.406710 2450 tablet_bootstrap.cc:492] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Bootstrap complete.
I20240503 07:59:54.406879 2450 ts_tablet_manager.cc:1367] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:54.407333 2450 raft_consensus.cc:348] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } }
I20240503 07:59:54.407441 2450 raft_consensus.cc:729] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fe8c6b1251e54f1e87923acb428eb2c8, State: Initialized, Role: LEARNER
I20240503 07:59:54.407539 2450 consensus_queue.cc:260] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: true } }
I20240503 07:59:54.407908 2450 ts_tablet_manager.cc:1398] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:54.407979 2431 heartbeater.cc:503] Master 127.0.99.62:44675 was elected leader, sending a full tablet report...
I20240503 07:59:54.408442 2206 tablet_copy_service.cc:342] P 66527bcbf3a34a158e1595e44b4ea248: Request end of tablet copy session fe8c6b1251e54f1e87923acb428eb2c8-1f4b086cb6a9411eac66da6d2506e590 received from {username='slave'} at 127.0.0.1:32828
I20240503 07:59:54.408529 2206 tablet_copy_service.cc:433] P 66527bcbf3a34a158e1595e44b4ea248: ending tablet copy session fe8c6b1251e54f1e87923acb428eb2c8-1f4b086cb6a9411eac66da6d2506e590 on tablet 1f4b086cb6a9411eac66da6d2506e590 with peer fe8c6b1251e54f1e87923acb428eb2c8
I20240503 07:59:54.808497 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:54.808952 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:54.809034 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:54.826179 2405 raft_consensus.cc:1203] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:55.264441 2460 raft_consensus.cc:1050] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: attempting to promote NON_VOTER fe8c6b1251e54f1e87923acb428eb2c8 to VOTER
I20240503 07:59:55.265022 2460 consensus_queue.cc:237] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } }
I20240503 07:59:55.266582 2405 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 LEARNER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:55.266702 2271 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:55.266582 2120 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:55.267084 2463 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:55.267428 2460 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:55.267872 2462 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:55.269660 2462 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.270076 2120 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.270159 2405 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.272670 2016 catalog_manager.cc:5484] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 reported cstate change: config changed from index 2 to 3, fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66527bcbf3a34a158e1595e44b4ea248" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.273494 2271 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fe8c6b1251e54f1e87923acb428eb2c8 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e48e9f07afa4101bd939f37a16cfbdb" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44483 } attrs { replace: true } } peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.277532 2196 consensus_queue.cc:237] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } }
I20240503 07:59:55.279043 2405 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.279637 2271 raft_consensus.cc:1261] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Refusing update from remote peer 66527bcbf3a34a158e1595e44b4ea248: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.279645 2439 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.280002 2439 consensus_queue.cc:1035] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.281414 2463 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.281674 2405 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.281939 2271 raft_consensus.cc:2932] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } } }
I20240503 07:59:55.283516 2002 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f4b086cb6a9411eac66da6d2506e590 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:55.283682 2014 catalog_manager.cc:5484] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 reported cstate change: config changed from index 3 to 4, VOTER 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "66527bcbf3a34a158e1595e44b4ea248" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66527bcbf3a34a158e1595e44b4ea248" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a25c2ccb594a4da9ce40ec51e3b149" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8c6b1251e54f1e87923acb428eb2c8" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 39737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:55.286527 2109 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4b086cb6a9411eac66da6d2506e590 with delete_type TABLET_DATA_TOMBSTONED (TS 6e48e9f07afa4101bd939f37a16cfbdb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50512
I20240503 07:59:55.287084 2472 tablet_replica.cc:330] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: stopping tablet replica
I20240503 07:59:55.287231 2472 raft_consensus.cc:2229] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:55.287366 2472 raft_consensus.cc:2258] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:55.287950 2472 ts_tablet_manager.cc:1875] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:55.289989 2472 ts_tablet_manager.cc:1888] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:55.290088 2472 log.cc:1196] T 1f4b086cb6a9411eac66da6d2506e590 P 6e48e9f07afa4101bd939f37a16cfbdb: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723166506157-396-0/minicluster-data/ts-0-root/wals/1f4b086cb6a9411eac66da6d2506e590
I20240503 07:59:55.290623 2001 catalog_manager.cc:4830] TS 6e48e9f07afa4101bd939f37a16cfbdb (127.0.99.1:44483): tablet 1f4b086cb6a9411eac66da6d2506e590 (table test-workload [id=91ae9fe55b954a4c8b7ca0738aa680d4]) successfully deleted
I20240503 07:59:56.809396 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:56.809772 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:56.809831 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 07:59:58.810103 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 07:59:58.810449 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 07:59:58.810509 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:00.810878 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:00.811241 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:00.811300 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:02.811646 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:02.812004 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:02.812059 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:04.812425 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:04.812800 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:04.812856 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:06.813153 2071 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:06.813560 2071 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:06.813632 2071 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:07.535962 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 08:00:07.539852 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:07.540103 396 tablet_replica.cc:330] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb: stopping tablet replica
I20240503 08:00:07.540239 396 raft_consensus.cc:2229] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:07.540370 396 raft_consensus.cc:2258] T 1d7a1a459ba4433da09a0a3e7efac6ea P 6e48e9f07afa4101bd939f37a16cfbdb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.553210 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 08:00:07.555227 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 08:00:07.559907 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:07.560092 396 tablet_replica.cc:330] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248: stopping tablet replica
I20240503 08:00:07.560246 396 raft_consensus.cc:2229] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:07.560429 396 raft_consensus.cc:2258] T 1f4b086cb6a9411eac66da6d2506e590 P 66527bcbf3a34a158e1595e44b4ea248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.560832 396 tablet_replica.cc:330] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248: stopping tablet replica
I20240503 08:00:07.560904 396 raft_consensus.cc:2229] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:07.560978 396 raft_consensus.cc:2258] T 1d7a1a459ba4433da09a0a3e7efac6ea P 66527bcbf3a34a158e1595e44b4ea248 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.573870 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 08:00:07.576009 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 08:00:07.580478 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:07.580685 396 tablet_replica.cc:330] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149: stopping tablet replica
I20240503 08:00:07.580801 396 raft_consensus.cc:2229] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:07.580950 396 raft_consensus.cc:2258] T 1f4b086cb6a9411eac66da6d2506e590 P 30a25c2ccb594a4da9ce40ec51e3b149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.581359 396 tablet_replica.cc:330] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149: stopping tablet replica
I20240503 08:00:07.581460 396 raft_consensus.cc:2229] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:00:07.581631 396 raft_consensus.cc:2258] T 1d7a1a459ba4433da09a0a3e7efac6ea P 30a25c2ccb594a4da9ce40ec51e3b149 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.595041 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 08:00:07.597260 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 08:00:07.601934 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:07.602138 396 tablet_replica.cc:330] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8: stopping tablet replica
I20240503 08:00:07.602258 396 raft_consensus.cc:2229] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:07.602399 396 raft_consensus.cc:2258] T 1f4b086cb6a9411eac66da6d2506e590 P fe8c6b1251e54f1e87923acb428eb2c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.615828 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 08:00:07.617812 396 master.cc:537] Master@127.0.99.62:44675 shutting down...
I20240503 08:00:07.621793 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:07.621965 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.622013 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 44bfdf1b07304b30920566b08f3bf8c1: stopping tablet replica
I20240503 08:00:07.634577 396 master.cc:552] Master@127.0.99.62:44675 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26886 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 08:00:07.642431 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:39623
I20240503 08:00:07.642763 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:07.645249 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.646385 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.647167 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:07.648082 396 server_base.cc:1034] running on GCE node
I20240503 08:00:07.648288 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:07.648329 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:07.648345 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723207648345 us; error 0 us; skew 500 ppm
I20240503 08:00:07.656522 396 webserver.cc:460] Webserver started at http://127.0.99.62:34429/ using document root <none> and password file <none>
I20240503 08:00:07.656682 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:07.656728 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:07.656790 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:07.657179 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "22b3b9775fd74a8d81a5e40be5788774"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-mxwh"
I20240503 08:00:07.658623 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:07.659519 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.659783 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:07.659852 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "22b3b9775fd74a8d81a5e40be5788774"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-mxwh"
I20240503 08:00:07.659926 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:07.680063 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:07.680492 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:07.684605 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:39623
I20240503 08:00:07.686928 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:39623 every 8 connection(s)
I20240503 08:00:07.687369 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.689145 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774: Bootstrap starting.
I20240503 08:00:07.689884 2536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.690827 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774: No bootstrap required, opened a new log
I20240503 08:00:07.691184 2536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER }
I20240503 08:00:07.691268 2536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.691290 2536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b3b9775fd74a8d81a5e40be5788774, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.691427 2536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER }
I20240503 08:00:07.691496 2536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:07.691533 2536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:07.691587 2536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.692265 2536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER }
I20240503 08:00:07.692381 2536 leader_election.cc:302] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b3b9775fd74a8d81a5e40be5788774; no voters:
I20240503 08:00:07.692593 2536 leader_election.cc:288] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:07.692705 2539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.692935 2539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 1 LEADER]: Becoming Leader. State: Replica: 22b3b9775fd74a8d81a5e40be5788774, State: Running, Role: LEADER
I20240503 08:00:07.693043 2536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:07.693087 2539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER }
I20240503 08:00:07.693707 2540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b3b9775fd74a8d81a5e40be5788774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER } }
I20240503 08:00:07.693722 2541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b3b9775fd74a8d81a5e40be5788774. Latest consensus state: current_term: 1 leader_uuid: "22b3b9775fd74a8d81a5e40be5788774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b3b9775fd74a8d81a5e40be5788774" member_type: VOTER } }
I20240503 08:00:07.693872 2541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:07.694090 2540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:07.694340 2548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:07.695127 2548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:07.695318 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:00:07.697014 2548 catalog_manager.cc:1347] Generated new cluster ID: 62b4098134c84d7c919c5de19deefd09
I20240503 08:00:07.697082 2548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:07.716087 2548 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:07.716745 2548 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:07.731758 2548 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774: Generated new TSK 0
I20240503 08:00:07.731951 2548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:07.759688 396 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.8487e-05s
I20240503 08:00:19.839094 396 master.cc:537] Master@127.0.99.62:39623 shutting down...
I20240503 08:00:19.842417 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:19.842589 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:19.842643 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22b3b9775fd74a8d81a5e40be5788774: stopping tablet replica
I20240503 08:00:19.854750 396 master.cc:552] Master@127.0.99.62:39623 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12216 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 08:00:19.858758 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:37331
I20240503 08:00:19.859068 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:19.861752 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.862679 2559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.863226 2558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:19.864112 396 server_base.cc:1034] running on GCE node
I20240503 08:00:19.864281 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:19.864320 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:19.864334 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723219864334 us; error 0 us; skew 500 ppm
I20240503 08:00:19.865270 396 webserver.cc:460] Webserver started at http://127.0.99.62:36635/ using document root <none> and password file <none>
I20240503 08:00:19.865415 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:19.865465 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:19.865578 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:19.865962 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "070f15fe20ca433e9a0e7cb73dc8eda2"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.867496 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:19.868556 2566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:19.868839 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:19.868918 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "070f15fe20ca433e9a0e7cb73dc8eda2"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.868999 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:19.874820 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:19.875167 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:19.878975 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:37331
I20240503 08:00:19.879557 2618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:37331 every 8 connection(s)
I20240503 08:00:19.879972 2619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:19.881723 2619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2: Bootstrap starting.
I20240503 08:00:19.882442 2619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:19.883397 2619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2: No bootstrap required, opened a new log
I20240503 08:00:19.883744 2619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER }
I20240503 08:00:19.883834 2619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:19.883868 2619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070f15fe20ca433e9a0e7cb73dc8eda2, State: Initialized, Role: FOLLOWER
I20240503 08:00:19.884021 2619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER }
I20240503 08:00:19.884092 2619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:19.884138 2619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:19.884213 2619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:19.884851 2619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER }
I20240503 08:00:19.884960 2619 leader_election.cc:302] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070f15fe20ca433e9a0e7cb73dc8eda2; no voters:
I20240503 08:00:19.885098 2619 leader_election.cc:288] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:19.885215 2622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:19.885394 2622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 1 LEADER]: Becoming Leader. State: Replica: 070f15fe20ca433e9a0e7cb73dc8eda2, State: Running, Role: LEADER
I20240503 08:00:19.885509 2619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:19.885522 2622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER }
I20240503 08:00:19.885977 2623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER } }
I20240503 08:00:19.886029 2624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f15fe20ca433e9a0e7cb73dc8eda2. Latest consensus state: current_term: 1 leader_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070f15fe20ca433e9a0e7cb73dc8eda2" member_type: VOTER } }
I20240503 08:00:19.886137 2623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:19.886224 2624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:19.886675 2630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:19.887519 2630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:19.887740 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:00:19.889261 2630 catalog_manager.cc:1347] Generated new cluster ID: 9661e16099f843d7b08436f6b164eb56
I20240503 08:00:19.889315 2630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:19.895903 2630 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:19.896479 2630 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:19.904701 2630 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2: Generated new TSK 0
I20240503 08:00:19.904881 2630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:19.920256 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:19.922824 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.923738 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.924558 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:19.925416 396 server_base.cc:1034] running on GCE node
I20240503 08:00:19.925617 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:19.925657 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:19.925673 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723219925673 us; error 0 us; skew 500 ppm
I20240503 08:00:19.926664 396 webserver.cc:460] Webserver started at http://127.0.99.1:35075/ using document root <none> and password file <none>
I20240503 08:00:19.926813 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:19.926863 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:19.926946 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:19.927345 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "ab5d22cf179044258bd3347bb997b3fe"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.928831 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:19.929757 2648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:19.930114 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:19.930200 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "ab5d22cf179044258bd3347bb997b3fe"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.930279 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:19.972821 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:19.973227 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:19.973532 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:19.973974 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:19.974013 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:19.974064 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:19.974098 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:19.978173 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:38595
I20240503 08:00:19.978205 2711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:38595 every 8 connection(s)
I20240503 08:00:19.979043 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:19.981417 2716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.982399 2719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:19.983278 2717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:19.984058 2712 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37331
I20240503 08:00:19.984107 396 server_base.cc:1034] running on GCE node
I20240503 08:00:19.984198 2712 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:19.984292 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:19.984334 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:19.984356 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723219984355 us; error 0 us; skew 500 ppm
I20240503 08:00:19.984453 2712 heartbeater.cc:511] Master 127.0.99.62:37331 requested a full tablet report, sending...
I20240503 08:00:19.985159 2583 ts_manager.cc:194] Registered new tserver with Master: ab5d22cf179044258bd3347bb997b3fe (127.0.99.1:38595)
I20240503 08:00:19.985316 396 webserver.cc:460] Webserver started at http://127.0.99.2:35373/ using document root <none> and password file <none>
I20240503 08:00:19.985455 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:19.985512 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:19.985601 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:19.986043 2583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43630
I20240503 08:00:19.986176 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "7993e0c126a7466a83121ffe6b01367c"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.987903 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:19.988925 2724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:19.989197 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:19.989264 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "7993e0c126a7466a83121ffe6b01367c"
format_stamp: "Formatted at 2024-05-03 08:00:19 on dist-test-slave-mxwh"
I20240503 08:00:19.989336 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:20.004060 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:20.004524 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:20.004842 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:20.005301 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:20.005344 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:20.005396 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:20.005426 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:20.010097 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:45683
I20240503 08:00:20.010161 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:45683 every 8 connection(s)
I20240503 08:00:20.011101 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:20.013830 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:20.014751 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:20.014981 2788 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37331
I20240503 08:00:20.015076 2788 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:20.015280 2788 heartbeater.cc:511] Master 127.0.99.62:37331 requested a full tablet report, sending...
I20240503 08:00:20.015890 2583 ts_manager.cc:194] Registered new tserver with Master: 7993e0c126a7466a83121ffe6b01367c (127.0.99.2:45683)
I20240503 08:00:20.015945 396 server_base.cc:1034] running on GCE node
I20240503 08:00:20.016510 2583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43636
W20240503 08:00:20.016782 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:20.017024 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:20.017076 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:20.017093 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723220017093 us; error 0 us; skew 500 ppm
I20240503 08:00:20.018144 396 webserver.cc:460] Webserver started at http://127.0.99.3:38115/ using document root <none> and password file <none>
I20240503 08:00:20.018298 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:20.018349 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:20.018414 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:20.018976 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "bfeb1ca00b3c4e2f93f4ae7a3113bb9b"
format_stamp: "Formatted at 2024-05-03 08:00:20 on dist-test-slave-mxwh"
I20240503 08:00:20.020562 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:20.021637 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:20.022070 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:20.022145 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "bfeb1ca00b3c4e2f93f4ae7a3113bb9b"
format_stamp: "Formatted at 2024-05-03 08:00:20 on dist-test-slave-mxwh"
I20240503 08:00:20.022228 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:20.029484 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:20.029877 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:20.030155 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:20.030627 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:20.030670 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:20.030715 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:20.030750 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:20.035706 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:38673
I20240503 08:00:20.035749 2862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:38673 every 8 connection(s)
I20240503 08:00:20.040531 2863 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37331
I20240503 08:00:20.040673 2863 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:20.040921 2863 heartbeater.cc:511] Master 127.0.99.62:37331 requested a full tablet report, sending...
I20240503 08:00:20.041582 2583 ts_manager.cc:194] Registered new tserver with Master: bfeb1ca00b3c4e2f93f4ae7a3113bb9b (127.0.99.3:38673)
I20240503 08:00:20.041785 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005666183s
I20240503 08:00:20.042295 2583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43640
I20240503 08:00:20.987108 2712 heartbeater.cc:503] Master 127.0.99.62:37331 was elected leader, sending a full tablet report...
I20240503 08:00:21.017509 2788 heartbeater.cc:503] Master 127.0.99.62:37331 was elected leader, sending a full tablet report...
I20240503 08:00:21.043320 2863 heartbeater.cc:503] Master 127.0.99.62:37331 was elected leader, sending a full tablet report...
I20240503 08:00:21.887555 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:23.887915 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:25.888339 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:27.888669 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:29.889030 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:31.889410 2637 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:32.129511 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 08:00:32.133816 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:32.146679 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 08:00:32.148629 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 08:00:32.152849 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:32.164985 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 08:00:32.166821 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 08:00:32.171048 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:32.183324 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 08:00:32.185285 396 master.cc:537] Master@127.0.99.62:37331 shutting down...
I20240503 08:00:32.188697 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:32.188874 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:32.188943 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 070f15fe20ca433e9a0e7cb73dc8eda2: stopping tablet replica
I20240503 08:00:32.201141 396 master.cc:552] Master@127.0.99.62:37331 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12347 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 08:00:32.206881 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:36079
I20240503 08:00:32.207206 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.209844 2868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.210826 2870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.211690 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.212641 396 server_base.cc:1034] running on GCE node
I20240503 08:00:32.212800 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:32.212882 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:32.212898 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723232212898 us; error 0 us; skew 500 ppm
I20240503 08:00:32.213905 396 webserver.cc:460] Webserver started at http://127.0.99.62:38621/ using document root <none> and password file <none>
I20240503 08:00:32.214054 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.214099 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.214177 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.214571 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "5b8ff2ef2f574879a3c97121b1dbc401"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.216266 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:32.217202 2875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.217423 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:32.217491 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "5b8ff2ef2f574879a3c97121b1dbc401"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.217562 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.238399 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.238781 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.242801 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:36079
I20240503 08:00:32.244006 2927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:36079 every 8 connection(s)
I20240503 08:00:32.244470 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:32.246181 2928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401: Bootstrap starting.
I20240503 08:00:32.246903 2928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:32.247831 2928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401: No bootstrap required, opened a new log
I20240503 08:00:32.248256 2928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER }
I20240503 08:00:32.248346 2928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:32.248381 2928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8ff2ef2f574879a3c97121b1dbc401, State: Initialized, Role: FOLLOWER
I20240503 08:00:32.248521 2928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER }
I20240503 08:00:32.248601 2928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:32.248649 2928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:32.248703 2928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:32.249346 2928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER }
I20240503 08:00:32.249466 2928 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8ff2ef2f574879a3c97121b1dbc401; no voters:
I20240503 08:00:32.249635 2928 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:32.249784 2931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:32.249979 2931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8ff2ef2f574879a3c97121b1dbc401, State: Running, Role: LEADER
I20240503 08:00:32.250164 2931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER }
I20240503 08:00:32.250236 2928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:32.250727 2932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER } }
I20240503 08:00:32.250772 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8ff2ef2f574879a3c97121b1dbc401. Latest consensus state: current_term: 1 leader_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8ff2ef2f574879a3c97121b1dbc401" member_type: VOTER } }
I20240503 08:00:32.250891 2932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:32.250909 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:32.251451 2938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:32.252094 2938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:32.252365 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:00:32.253906 2938 catalog_manager.cc:1347] Generated new cluster ID: 850ee4e642fd4aa99fafc1d26589821a
I20240503 08:00:32.253971 2938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:32.259972 2938 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:32.260591 2938 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:32.272460 2938 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401: Generated new TSK 0
I20240503 08:00:32.272692 2938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:32.284934 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.288110 2950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.289060 2952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.289626 2949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.290499 396 server_base.cc:1034] running on GCE node
I20240503 08:00:32.290691 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:32.290735 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:32.290750 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723232290750 us; error 0 us; skew 500 ppm
I20240503 08:00:32.291712 396 webserver.cc:460] Webserver started at http://127.0.99.1:38597/ using document root <none> and password file <none>
I20240503 08:00:32.291858 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.291906 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.291961 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.292397 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "2a589f013e6c4e079c5e980e54e848ff"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.293850 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:32.294819 2957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.295082 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:32.295151 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "2a589f013e6c4e079c5e980e54e848ff"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.295219 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.318995 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.319363 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.319604 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.320087 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.320127 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.320181 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.320226 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.324030 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43569
I20240503 08:00:32.324110 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43569 every 8 connection(s)
I20240503 08:00:32.324841 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.327791 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.328354 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.329335 396 server_base.cc:1034] running on GCE node
I20240503 08:00:32.329413 3021 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36079
I20240503 08:00:32.329499 3021 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.329725 3021 heartbeater.cc:511] Master 127.0.99.62:36079 requested a full tablet report, sending...
W20240503 08:00:32.330317 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.330430 2892 ts_manager.cc:194] Registered new tserver with Master: 2a589f013e6c4e079c5e980e54e848ff (127.0.99.1:43569)
I20240503 08:00:32.330612 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:32.330668 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:32.330693 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723232330692 us; error 0 us; skew 500 ppm
I20240503 08:00:32.331193 2892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56618
I20240503 08:00:32.331866 396 webserver.cc:460] Webserver started at http://127.0.99.2:38631/ using document root <none> and password file <none>
I20240503 08:00:32.332048 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.332115 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.332288 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.332756 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "1eed346a3f7c4845a78956e4a2001c98"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.334254 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:32.335258 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.335518 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:32.335602 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "1eed346a3f7c4845a78956e4a2001c98"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.335697 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.353067 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.353466 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.353766 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.354218 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.354260 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.354308 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.354343 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.358548 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:36769
I20240503 08:00:32.358608 3096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:36769 every 8 connection(s)
I20240503 08:00:32.359378 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.361905 3100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.362871 3101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.363826 3103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.364046 3097 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36079
I20240503 08:00:32.364135 3097 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.364377 3097 heartbeater.cc:511] Master 127.0.99.62:36079 requested a full tablet report, sending...
I20240503 08:00:32.364962 396 server_base.cc:1034] running on GCE node
I20240503 08:00:32.365072 2892 ts_manager.cc:194] Registered new tserver with Master: 1eed346a3f7c4845a78956e4a2001c98 (127.0.99.2:36769)
I20240503 08:00:32.365173 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:32.365228 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:32.365254 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723232365252 us; error 0 us; skew 500 ppm
I20240503 08:00:32.365649 2892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56630
I20240503 08:00:32.366320 396 webserver.cc:460] Webserver started at http://127.0.99.3:37205/ using document root <none> and password file <none>
I20240503 08:00:32.366485 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.366545 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.366601 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.366966 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "34cdc845a9c045929510123786ad860d"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.368371 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:32.369441 3108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.369763 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:32.369830 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "34cdc845a9c045929510123786ad860d"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.369904 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.383553 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.383952 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.384310 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.384791 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.384831 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.384863 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.384878 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.389272 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:37087
I20240503 08:00:32.389319 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:37087 every 8 connection(s)
I20240503 08:00:32.390044 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.393150 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.394214 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.394663 3172 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36079
I20240503 08:00:32.394876 3172 heartbeater.cc:464] Registering TS with master...
W20240503 08:00:32.395110 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.395205 3172 heartbeater.cc:511] Master 127.0.99.62:36079 requested a full tablet report, sending...
I20240503 08:00:32.396035 2892 ts_manager.cc:194] Registered new tserver with Master: 34cdc845a9c045929510123786ad860d (127.0.99.3:37087)
I20240503 08:00:32.396857 2892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56640
I20240503 08:00:32.397338 396 server_base.cc:1034] running on GCE node
I20240503 08:00:32.397580 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:32.397639 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:32.397671 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723232397670 us; error 0 us; skew 500 ppm
I20240503 08:00:32.398722 396 webserver.cc:460] Webserver started at http://127.0.99.4:44025/ using document root <none> and password file <none>
I20240503 08:00:32.398856 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.398905 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.398969 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.399318 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-3-root/instance:
uuid: "20f1d44e710d49d7ab67cbf7ba6496f9"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.400777 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:32.401672 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.401942 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:32.402019 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-3-root
uuid: "20f1d44e710d49d7ab67cbf7ba6496f9"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-mxwh"
I20240503 08:00:32.402088 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723166506157-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.408809 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.409214 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.409510 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.409960 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.409999 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.410049 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.410090 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.414917 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:39187
I20240503 08:00:32.414973 3246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:39187 every 8 connection(s)
I20240503 08:00:32.419421 3247 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36079
I20240503 08:00:32.419517 3247 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.419731 3247 heartbeater.cc:511] Master 127.0.99.62:36079 requested a full tablet report, sending...
I20240503 08:00:32.420351 2892 ts_manager.cc:194] Registered new tserver with Master: 20f1d44e710d49d7ab67cbf7ba6496f9 (127.0.99.4:39187)
I20240503 08:00:32.420876 2892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56642
I20240503 08:00:32.421038 396 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.005701994s
I20240503 08:00:33.332343 3021 heartbeater.cc:503] Master 127.0.99.62:36079 was elected leader, sending a full tablet report...
I20240503 08:00:33.366662 3097 heartbeater.cc:503] Master 127.0.99.62:36079 was elected leader, sending a full tablet report...
I20240503 08:00:33.397996 3172 heartbeater.cc:503] Master 127.0.99.62:36079 was elected leader, sending a full tablet report...
I20240503 08:00:33.421805 3247 heartbeater.cc:503] Master 127.0.99.62:36079 was elected leader, sending a full tablet report...
I20240503 08:00:33.452242 396 test_util.cc:268] Using random seed: 490197198
I20240503 08:00:33.460266 2892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56660:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:00:33.470564 3061 tablet_service.cc:1467] Processing CreateTablet for tablet fa7a46fc7c74499b966dc6784895ebc1 (DEFAULT_TABLE table=test-workload [id=4f6bc8238f7d4f5e8a4bb759d63ead36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:33.470762 3136 tablet_service.cc:1467] Processing CreateTablet for tablet fa7a46fc7c74499b966dc6784895ebc1 (DEFAULT_TABLE table=test-workload [id=4f6bc8238f7d4f5e8a4bb759d63ead36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:33.470911 3061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7a46fc7c74499b966dc6784895ebc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:33.470868 2985 tablet_service.cc:1467] Processing CreateTablet for tablet fa7a46fc7c74499b966dc6784895ebc1 (DEFAULT_TABLE table=test-workload [id=4f6bc8238f7d4f5e8a4bb759d63ead36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:33.471011 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7a46fc7c74499b966dc6784895ebc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:33.471081 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7a46fc7c74499b966dc6784895ebc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:33.473475 3270 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: Bootstrap starting.
I20240503 08:00:33.474277 3270 tablet_bootstrap.cc:654] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:33.474839 3269 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: Bootstrap starting.
I20240503 08:00:33.475345 3270 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: No bootstrap required, opened a new log
I20240503 08:00:33.475419 3270 ts_tablet_manager.cc:1367] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:33.475581 3269 tablet_bootstrap.cc:654] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:33.475837 3270 raft_consensus.cc:348] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.475956 3270 raft_consensus.cc:374] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:33.476003 3270 raft_consensus.cc:729] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eed346a3f7c4845a78956e4a2001c98, State: Initialized, Role: FOLLOWER
I20240503 08:00:33.476173 3268 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: Bootstrap starting.
I20240503 08:00:33.476140 3270 consensus_queue.cc:260] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.476675 3269 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: No bootstrap required, opened a new log
I20240503 08:00:33.476688 3270 ts_tablet_manager.cc:1398] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:33.476773 3269 ts_tablet_manager.cc:1367] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:33.477211 3269 raft_consensus.cc:348] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.477315 3269 raft_consensus.cc:374] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:33.477355 3269 raft_consensus.cc:729] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a589f013e6c4e079c5e980e54e848ff, State: Initialized, Role: FOLLOWER
I20240503 08:00:33.477476 3269 consensus_queue.cc:260] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.477591 3268 tablet_bootstrap.cc:654] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:33.477855 3269 ts_tablet_manager.cc:1398] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:33.478756 3268 tablet_bootstrap.cc:492] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: No bootstrap required, opened a new log
I20240503 08:00:33.478847 3268 ts_tablet_manager.cc:1367] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:33.479290 3268 raft_consensus.cc:348] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.479401 3268 raft_consensus.cc:374] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:33.479454 3268 raft_consensus.cc:729] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cdc845a9c045929510123786ad860d, State: Initialized, Role: FOLLOWER
I20240503 08:00:33.479590 3268 consensus_queue.cc:260] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.479954 3268 ts_tablet_manager.cc:1398] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 08:00:33.575857 3022 tablet.cc:2369] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:33.595404 3276 raft_consensus.cc:482] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:33.595672 3276 raft_consensus.cc:504] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.596302 3276 leader_election.cc:288] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eed346a3f7c4845a78956e4a2001c98 (127.0.99.2:36769), 2a589f013e6c4e079c5e980e54e848ff (127.0.99.1:43569)
I20240503 08:00:33.599712 3071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7a46fc7c74499b966dc6784895ebc1" candidate_uuid: "34cdc845a9c045929510123786ad860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eed346a3f7c4845a78956e4a2001c98" is_pre_election: true
I20240503 08:00:33.599752 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7a46fc7c74499b966dc6784895ebc1" candidate_uuid: "34cdc845a9c045929510123786ad860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a589f013e6c4e079c5e980e54e848ff" is_pre_election: true
I20240503 08:00:33.599858 3071 raft_consensus.cc:2454] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cdc845a9c045929510123786ad860d in term 0.
I20240503 08:00:33.599866 2995 raft_consensus.cc:2454] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cdc845a9c045929510123786ad860d in term 0.
I20240503 08:00:33.600167 3112 leader_election.cc:302] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eed346a3f7c4845a78956e4a2001c98, 34cdc845a9c045929510123786ad860d; no voters:
I20240503 08:00:33.600348 3276 raft_consensus.cc:2781] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:33.600402 3276 raft_consensus.cc:482] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:33.600440 3276 raft_consensus.cc:3037] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:33.601243 3276 raft_consensus.cc:504] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.601537 3276 leader_election.cc:288] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [CANDIDATE]: Term 1 election: Requested vote from peers 1eed346a3f7c4845a78956e4a2001c98 (127.0.99.2:36769), 2a589f013e6c4e079c5e980e54e848ff (127.0.99.1:43569)
I20240503 08:00:33.601864 3071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7a46fc7c74499b966dc6784895ebc1" candidate_uuid: "34cdc845a9c045929510123786ad860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eed346a3f7c4845a78956e4a2001c98"
I20240503 08:00:33.601864 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7a46fc7c74499b966dc6784895ebc1" candidate_uuid: "34cdc845a9c045929510123786ad860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a589f013e6c4e079c5e980e54e848ff"
I20240503 08:00:33.601974 3071 raft_consensus.cc:3037] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:33.601986 2995 raft_consensus.cc:3037] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:33.602906 3071 raft_consensus.cc:2454] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cdc845a9c045929510123786ad860d in term 1.
I20240503 08:00:33.602924 2995 raft_consensus.cc:2454] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cdc845a9c045929510123786ad860d in term 1.
I20240503 08:00:33.603168 3111 leader_election.cc:302] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a589f013e6c4e079c5e980e54e848ff, 34cdc845a9c045929510123786ad860d; no voters:
I20240503 08:00:33.603399 3276 raft_consensus.cc:2781] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:33.603597 3276 raft_consensus.cc:686] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 1 LEADER]: Becoming Leader. State: Replica: 34cdc845a9c045929510123786ad860d, State: Running, Role: LEADER
I20240503 08:00:33.603739 3276 consensus_queue.cc:237] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } }
I20240503 08:00:33.605195 2891 catalog_manager.cc:5484] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d reported cstate change: term changed from 0 to 1, leader changed from <none> to 34cdc845a9c045929510123786ad860d (127.0.99.3). New cstate: current_term: 1 leader_uuid: "34cdc845a9c045929510123786ad860d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cdc845a9c045929510123786ad860d" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:33.981525 3276 consensus_queue.cc:1035] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eed346a3f7c4845a78956e4a2001c98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:33.984862 3276 consensus_queue.cc:1035] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a589f013e6c4e079c5e980e54e848ff" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:34.252373 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:34.252724 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:34.252806 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:36.253144 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:36.253435 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:36.253490 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:38.253799 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:38.254094 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:38.254146 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:40.254427 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:40.254711 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:40.254765 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:42.255043 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:42.255319 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:42.255373 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:44.255699 2946 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:44.256014 2946 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:44.256084 2946 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:44.669915 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 08:00:44.673408 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:44.673625 396 tablet_replica.cc:330] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff: stopping tablet replica
I20240503 08:00:44.673748 396 raft_consensus.cc:2229] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.673877 396 raft_consensus.cc:2258] T fa7a46fc7c74499b966dc6784895ebc1 P 2a589f013e6c4e079c5e980e54e848ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.686554 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 08:00:44.688694 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 08:00:44.692436 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:44.692692 396 tablet_replica.cc:330] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98: stopping tablet replica
I20240503 08:00:44.692811 396 raft_consensus.cc:2229] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.692940 396 raft_consensus.cc:2258] T fa7a46fc7c74499b966dc6784895ebc1 P 1eed346a3f7c4845a78956e4a2001c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.705472 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 08:00:44.707494 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 08:00:44.710994 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:44.711171 396 tablet_replica.cc:330] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d: stopping tablet replica
I20240503 08:00:44.711285 396 raft_consensus.cc:2229] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:44.711469 396 raft_consensus.cc:2258] T fa7a46fc7c74499b966dc6784895ebc1 P 34cdc845a9c045929510123786ad860d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.724069 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 08:00:44.725941 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 08:00:44.729260 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:44.741079 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 08:00:44.742951 396 master.cc:537] Master@127.0.99.62:36079 shutting down...
I20240503 08:00:44.745733 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:44.745889 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.745954 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5b8ff2ef2f574879a3c97121b1dbc401: stopping tablet replica
I20240503 08:00:44.758158 396 master.cc:552] Master@127.0.99.62:36079 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12557 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 08:00:44.764496 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:35903
I20240503 08:00:44.764813 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:44.767391 3292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.768353 3295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.769315 396 server_base.cc:1034] running on GCE node
W20240503 08:00:44.770051 3293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.770270 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:44.770314 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:44.770329 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723244770329 us; error 0 us; skew 500 ppm
I20240503 08:00:44.771201 396 webserver.cc:460] Webserver started at http://127.0.99.62:46037/ using document root <none> and password file <none>
I20240503 08:00:44.771332 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:44.771369 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:44.771418 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:44.771776 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "bf764a419caf4ccdbe06794e28f03cd3"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.773269 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:44.774187 3300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.774434 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:44.774494 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "bf764a419caf4ccdbe06794e28f03cd3"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.774572 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:44.802369 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:44.802759 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:44.806703 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:35903
I20240503 08:00:44.809285 3352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:35903 every 8 connection(s)
I20240503 08:00:44.809682 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:44.811409 3353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3: Bootstrap starting.
I20240503 08:00:44.812129 3353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:44.813139 3353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3: No bootstrap required, opened a new log
I20240503 08:00:44.813464 3353 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER }
I20240503 08:00:44.813547 3353 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:44.813570 3353 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf764a419caf4ccdbe06794e28f03cd3, State: Initialized, Role: FOLLOWER
I20240503 08:00:44.813750 3353 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER }
I20240503 08:00:44.813858 3353 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:44.813889 3353 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:44.813931 3353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:44.814584 3353 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER }
I20240503 08:00:44.814695 3353 leader_election.cc:302] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf764a419caf4ccdbe06794e28f03cd3; no voters:
I20240503 08:00:44.814854 3353 leader_election.cc:288] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:44.815006 3356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:44.815214 3356 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 1 LEADER]: Becoming Leader. State: Replica: bf764a419caf4ccdbe06794e28f03cd3, State: Running, Role: LEADER
I20240503 08:00:44.815266 3353 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:44.815344 3356 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER }
I20240503 08:00:44.815850 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf764a419caf4ccdbe06794e28f03cd3. Latest consensus state: current_term: 1 leader_uuid: "bf764a419caf4ccdbe06794e28f03cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER } }
I20240503 08:00:44.815815 3357 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf764a419caf4ccdbe06794e28f03cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf764a419caf4ccdbe06794e28f03cd3" member_type: VOTER } }
I20240503 08:00:44.815958 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:44.816054 3357 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:44.816301 3364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:44.816962 3364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:44.817219 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:00:44.818696 3364 catalog_manager.cc:1347] Generated new cluster ID: 6e6ae47365fe48c091136e64272331b1
I20240503 08:00:44.818753 3364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:44.833768 3364 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:44.834303 3364 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:44.847224 3364 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3: Generated new TSK 0
I20240503 08:00:44.847407 3364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:44.849642 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:44.852614 3377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.853453 396 server_base.cc:1034] running on GCE node
W20240503 08:00:44.854185 3374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.854691 3375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.854876 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:44.854918 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:44.854933 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723244854933 us; error 0 us; skew 500 ppm
I20240503 08:00:44.855872 396 webserver.cc:460] Webserver started at http://127.0.99.1:39751/ using document root <none> and password file <none>
I20240503 08:00:44.856004 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:44.856048 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:44.856109 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:44.856501 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "b5a26c7225564c3d983917024268e307"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.857954 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:44.858881 3382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.859158 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:44.859230 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "b5a26c7225564c3d983917024268e307"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.859303 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:44.889231 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:44.889616 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:44.889921 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:44.890393 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:44.890452 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.890501 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:44.890525 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.894757 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43867
I20240503 08:00:44.894785 3445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43867 every 8 connection(s)
I20240503 08:00:44.895558 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:44.898259 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.899183 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.900172 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.900261 3446 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35903
I20240503 08:00:44.900372 3446 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:44.900595 3446 heartbeater.cc:511] Master 127.0.99.62:35903 requested a full tablet report, sending...
I20240503 08:00:44.900918 396 server_base.cc:1034] running on GCE node
I20240503 08:00:44.901091 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:44.901142 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:44.901162 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723244901162 us; error 0 us; skew 500 ppm
I20240503 08:00:44.901319 3317 ts_manager.cc:194] Registered new tserver with Master: b5a26c7225564c3d983917024268e307 (127.0.99.1:43867)
I20240503 08:00:44.902218 396 webserver.cc:460] Webserver started at http://127.0.99.2:36887/ using document root <none> and password file <none>
I20240503 08:00:44.902267 3317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46634
I20240503 08:00:44.902384 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:44.902446 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:44.902523 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:44.902920 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "63489f1a14744d0f8394e263d3ab4819"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.904376 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:44.905340 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.905570 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:44.905640 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "63489f1a14744d0f8394e263d3ab4819"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.905718 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:44.918514 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:44.918936 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:44.919234 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:44.919667 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:44.919705 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.919754 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:44.919785 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.923978 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:33029
I20240503 08:00:44.924012 3521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:33029 every 8 connection(s)
I20240503 08:00:44.924795 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:44.927263 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.928108 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:44.929064 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:44.929688 3522 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35903
I20240503 08:00:44.929793 3522 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:44.929899 396 server_base.cc:1034] running on GCE node
I20240503 08:00:44.930040 3522 heartbeater.cc:511] Master 127.0.99.62:35903 requested a full tablet report, sending...
I20240503 08:00:44.930040 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:44.930131 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:44.930161 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723244930160 us; error 0 us; skew 500 ppm
I20240503 08:00:44.930740 3317 ts_manager.cc:194] Registered new tserver with Master: 63489f1a14744d0f8394e263d3ab4819 (127.0.99.2:33029)
I20240503 08:00:44.931183 396 webserver.cc:460] Webserver started at http://127.0.99.3:46157/ using document root <none> and password file <none>
I20240503 08:00:44.931313 3317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46650
I20240503 08:00:44.931344 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:44.931416 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:44.931493 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:44.931983 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "e72b070143624ee7b2e011e32ddea5ca"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.933661 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:44.934585 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.934926 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:44.935007 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "e72b070143624ee7b2e011e32ddea5ca"
format_stamp: "Formatted at 2024-05-03 08:00:44 on dist-test-slave-mxwh"
I20240503 08:00:44.935087 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:44.941643 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:44.942059 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:44.942341 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:44.942806 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:44.942847 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.942898 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:44.942929 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:44.947628 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:41811
I20240503 08:00:44.947702 3596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:41811 every 8 connection(s)
I20240503 08:00:44.953845 3597 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35903
I20240503 08:00:44.953977 3597 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:44.954241 3597 heartbeater.cc:511] Master 127.0.99.62:35903 requested a full tablet report, sending...
I20240503 08:00:44.954943 3317 ts_manager.cc:194] Registered new tserver with Master: e72b070143624ee7b2e011e32ddea5ca (127.0.99.3:41811)
I20240503 08:00:44.955196 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006876904s
I20240503 08:00:44.955658 3317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46662
I20240503 08:00:45.903231 3446 heartbeater.cc:503] Master 127.0.99.62:35903 was elected leader, sending a full tablet report...
I20240503 08:00:45.932322 3522 heartbeater.cc:503] Master 127.0.99.62:35903 was elected leader, sending a full tablet report...
I20240503 08:00:45.956801 3597 heartbeater.cc:503] Master 127.0.99.62:35903 was elected leader, sending a full tablet report...
I20240503 08:00:45.986038 396 test_util.cc:268] Using random seed: 502730995
I20240503 08:00:45.993539 3317 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46688:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:00:45.993925 3317 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:00:46.002781 3410 tablet_service.cc:1467] Processing CreateTablet for tablet 80f912d3ed6b4dde94f16ff0d28bae98 (DEFAULT_TABLE table=test-workload [id=751e07e4d6ac41248d886b1124110e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:46.002923 3486 tablet_service.cc:1467] Processing CreateTablet for tablet 80f912d3ed6b4dde94f16ff0d28bae98 (DEFAULT_TABLE table=test-workload [id=751e07e4d6ac41248d886b1124110e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:46.003090 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f912d3ed6b4dde94f16ff0d28bae98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:46.003181 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f912d3ed6b4dde94f16ff0d28bae98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:46.003744 3561 tablet_service.cc:1467] Processing CreateTablet for tablet 80f912d3ed6b4dde94f16ff0d28bae98 (DEFAULT_TABLE table=test-workload [id=751e07e4d6ac41248d886b1124110e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:46.003962 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f912d3ed6b4dde94f16ff0d28bae98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:46.005239 3618 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: Bootstrap starting.
I20240503 08:00:46.005494 3619 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: Bootstrap starting.
I20240503 08:00:46.006209 3618 tablet_bootstrap.cc:654] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:46.006400 3619 tablet_bootstrap.cc:654] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:46.006420 3620 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: Bootstrap starting.
I20240503 08:00:46.007300 3620 tablet_bootstrap.cc:654] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:46.007570 3618 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: No bootstrap required, opened a new log
I20240503 08:00:46.007668 3619 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: No bootstrap required, opened a new log
I20240503 08:00:46.007704 3618 ts_tablet_manager.cc:1367] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:46.007756 3619 ts_tablet_manager.cc:1367] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:46.008203 3618 raft_consensus.cc:348] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.008224 3619 raft_consensus.cc:348] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.008311 3618 raft_consensus.cc:374] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:46.008323 3619 raft_consensus.cc:374] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:46.008354 3618 raft_consensus.cc:729] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63489f1a14744d0f8394e263d3ab4819, State: Initialized, Role: FOLLOWER
I20240503 08:00:46.008361 3619 raft_consensus.cc:729] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a26c7225564c3d983917024268e307, State: Initialized, Role: FOLLOWER
I20240503 08:00:46.008488 3620 tablet_bootstrap.cc:492] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: No bootstrap required, opened a new log
I20240503 08:00:46.008491 3618 consensus_queue.cc:260] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.008641 3620 ts_tablet_manager.cc:1367] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:46.008903 3618 ts_tablet_manager.cc:1398] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:46.008530 3619 consensus_queue.cc:260] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.009372 3619 ts_tablet_manager.cc:1398] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:46.009053 3620 raft_consensus.cc:348] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.009560 3620 raft_consensus.cc:374] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:46.009635 3620 raft_consensus.cc:729] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72b070143624ee7b2e011e32ddea5ca, State: Initialized, Role: FOLLOWER
I20240503 08:00:46.009791 3620 consensus_queue.cc:260] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.010143 3620 ts_tablet_manager.cc:1398] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:46.228607 3625 raft_consensus.cc:482] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:46.228801 3625 raft_consensus.cc:504] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.229346 3625 leader_election.cc:288] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63489f1a14744d0f8394e263d3ab4819 (127.0.99.2:33029), e72b070143624ee7b2e011e32ddea5ca (127.0.99.3:41811)
I20240503 08:00:46.233758 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f912d3ed6b4dde94f16ff0d28bae98" candidate_uuid: "b5a26c7225564c3d983917024268e307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72b070143624ee7b2e011e32ddea5ca" is_pre_election: true
I20240503 08:00:46.233915 3571 raft_consensus.cc:2454] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a26c7225564c3d983917024268e307 in term 0.
I20240503 08:00:46.234037 3496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f912d3ed6b4dde94f16ff0d28bae98" candidate_uuid: "b5a26c7225564c3d983917024268e307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63489f1a14744d0f8394e263d3ab4819" is_pre_election: true
I20240503 08:00:46.234189 3496 raft_consensus.cc:2454] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a26c7225564c3d983917024268e307 in term 0.
I20240503 08:00:46.234241 3383 leader_election.cc:302] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5a26c7225564c3d983917024268e307, e72b070143624ee7b2e011e32ddea5ca; no voters:
I20240503 08:00:46.234411 3625 raft_consensus.cc:2781] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:46.234470 3625 raft_consensus.cc:482] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:46.234513 3625 raft_consensus.cc:3037] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:46.235391 3625 raft_consensus.cc:504] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.235713 3625 leader_election.cc:288] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [CANDIDATE]: Term 1 election: Requested vote from peers 63489f1a14744d0f8394e263d3ab4819 (127.0.99.2:33029), e72b070143624ee7b2e011e32ddea5ca (127.0.99.3:41811)
I20240503 08:00:46.235976 3496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f912d3ed6b4dde94f16ff0d28bae98" candidate_uuid: "b5a26c7225564c3d983917024268e307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63489f1a14744d0f8394e263d3ab4819"
I20240503 08:00:46.236078 3496 raft_consensus.cc:3037] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:46.236876 3496 raft_consensus.cc:2454] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a26c7225564c3d983917024268e307 in term 1.
I20240503 08:00:46.237047 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f912d3ed6b4dde94f16ff0d28bae98" candidate_uuid: "b5a26c7225564c3d983917024268e307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72b070143624ee7b2e011e32ddea5ca"
I20240503 08:00:46.237131 3571 raft_consensus.cc:3037] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:46.237133 3385 leader_election.cc:302] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63489f1a14744d0f8394e263d3ab4819, b5a26c7225564c3d983917024268e307; no voters:
I20240503 08:00:46.237365 3625 raft_consensus.cc:2781] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:46.237569 3625 raft_consensus.cc:686] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 1 LEADER]: Becoming Leader. State: Replica: b5a26c7225564c3d983917024268e307, State: Running, Role: LEADER
I20240503 08:00:46.237706 3625 consensus_queue.cc:237] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } }
I20240503 08:00:46.238112 3571 raft_consensus.cc:2454] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a26c7225564c3d983917024268e307 in term 1.
I20240503 08:00:46.239307 3315 catalog_manager.cc:5484] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a26c7225564c3d983917024268e307 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "b5a26c7225564c3d983917024268e307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a26c7225564c3d983917024268e307" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:46.775547 3631 consensus_queue.cc:1035] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72b070143624ee7b2e011e32ddea5ca" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:46.778825 3634 consensus_queue.cc:1035] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63489f1a14744d0f8394e263d3ab4819" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:46.817427 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:46.817749 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:46.817839 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:46.817914 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:47.819577 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:48.818167 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:48.818457 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:48.818516 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:48.821095 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:49.822643 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:50.818804 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:50.819097 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:50.819154 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:50.824075 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:51.825578 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:52.819476 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:52.819801 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:52.819864 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:52.827212 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:53.828792 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:54.820187 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:54.820473 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:54.820526 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:54.830385 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:55.831945 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:56.820925 3371 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:56.821234 3371 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:00:56.821290 3371 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:00:56.833513 3365 placement_policy_util.cc:407] tablet 80f912d3ed6b4dde94f16ff0d28bae98: detected majority of replicas (2 of 3) at location L0
I20240503 08:00:57.342031 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 08:00:57.345616 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:57.345824 396 tablet_replica.cc:330] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307: stopping tablet replica
I20240503 08:00:57.345934 396 raft_consensus.cc:2229] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:57.346132 396 raft_consensus.cc:2258] T 80f912d3ed6b4dde94f16ff0d28bae98 P b5a26c7225564c3d983917024268e307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:57.358886 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 08:00:57.360788 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 08:00:57.364706 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:57.364921 396 tablet_replica.cc:330] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819: stopping tablet replica
I20240503 08:00:57.365032 396 raft_consensus.cc:2229] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:57.365154 396 raft_consensus.cc:2258] T 80f912d3ed6b4dde94f16ff0d28bae98 P 63489f1a14744d0f8394e263d3ab4819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:57.377753 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 08:00:57.379743 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 08:00:57.383460 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:00:57.383666 396 tablet_replica.cc:330] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca: stopping tablet replica
I20240503 08:00:57.383806 396 raft_consensus.cc:2229] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:57.383920 396 raft_consensus.cc:2258] T 80f912d3ed6b4dde94f16ff0d28bae98 P e72b070143624ee7b2e011e32ddea5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:57.396696 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 08:00:57.398591 396 master.cc:537] Master@127.0.99.62:35903 shutting down...
I20240503 08:00:57.401859 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:00:57.402032 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:57.402107 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P bf764a419caf4ccdbe06794e28f03cd3: stopping tablet replica
I20240503 08:00:57.414463 396 master.cc:552] Master@127.0.99.62:35903 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12656 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 08:00:57.420637 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:44179
I20240503 08:00:57.420954 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:00:57.424196 396 server_base.cc:1034] running on GCE node
W20240503 08:00:57.425024 3644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.425906 3642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.426419 3641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.426615 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:57.426703 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:57.426740 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723257426738 us; error 0 us; skew 500 ppm
I20240503 08:00:57.433506 396 webserver.cc:460] Webserver started at http://127.0.99.62:35777/ using document root <none> and password file <none>
I20240503 08:00:57.433686 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:57.433740 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:57.433848 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:57.434250 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/master-0-root/instance:
uuid: "caba1acbdc9b4bfe9593747f0a50da5f"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.435724 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:57.436712 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.436925 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:57.436995 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/master-0-root
uuid: "caba1acbdc9b4bfe9593747f0a50da5f"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.437063 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:57.455518 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:57.455897 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:57.459877 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:44179
I20240503 08:00:57.461678 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:44179 every 8 connection(s)
I20240503 08:00:57.462225 3702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:57.463958 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f: Bootstrap starting.
I20240503 08:00:57.464694 3702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:57.465620 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f: No bootstrap required, opened a new log
I20240503 08:00:57.465974 3702 raft_consensus.cc:348] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER }
I20240503 08:00:57.466058 3702 raft_consensus.cc:374] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:57.466091 3702 raft_consensus.cc:729] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caba1acbdc9b4bfe9593747f0a50da5f, State: Initialized, Role: FOLLOWER
I20240503 08:00:57.466217 3702 consensus_queue.cc:260] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER }
I20240503 08:00:57.466280 3702 raft_consensus.cc:388] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:57.466321 3702 raft_consensus.cc:482] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:57.466369 3702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:57.467010 3702 raft_consensus.cc:504] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER }
I20240503 08:00:57.467123 3702 leader_election.cc:302] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caba1acbdc9b4bfe9593747f0a50da5f; no voters:
I20240503 08:00:57.467288 3702 leader_election.cc:288] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:57.467375 3705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:57.467578 3705 raft_consensus.cc:686] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [term 1 LEADER]: Becoming Leader. State: Replica: caba1acbdc9b4bfe9593747f0a50da5f, State: Running, Role: LEADER
I20240503 08:00:57.467684 3702 sys_catalog.cc:564] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:57.467702 3705 consensus_queue.cc:237] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER }
I20240503 08:00:57.468127 3706 sys_catalog.cc:455] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER } }
I20240503 08:00:57.468170 3707 sys_catalog.cc:455] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader caba1acbdc9b4bfe9593747f0a50da5f. Latest consensus state: current_term: 1 leader_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caba1acbdc9b4bfe9593747f0a50da5f" member_type: VOTER } }
I20240503 08:00:57.468324 3706 sys_catalog.cc:458] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:57.468360 3707 sys_catalog.cc:458] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:57.468923 3711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:57.469566 3711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:57.469767 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:00:57.471159 3711 catalog_manager.cc:1347] Generated new cluster ID: 8383867f9dcb44bda59c541182d7f841
I20240503 08:00:57.471212 3711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:57.481681 3711 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:57.482256 3711 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:57.491761 3711 catalog_manager.cc:5857] T 00000000000000000000000000000000 P caba1acbdc9b4bfe9593747f0a50da5f: Generated new TSK 0
I20240503 08:00:57.491940 3711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:57.502077 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:57.504930 3726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.505880 3723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.506455 3724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.507364 396 server_base.cc:1034] running on GCE node
I20240503 08:00:57.507556 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:57.507597 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:57.507615 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723257507614 us; error 0 us; skew 500 ppm
I20240503 08:00:57.508637 396 webserver.cc:460] Webserver started at http://127.0.99.1:33675/ using document root <none> and password file <none>
I20240503 08:00:57.508812 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:57.508873 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:57.508955 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:57.509382 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root/instance:
uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.510908 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:57.511833 3731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.512073 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:57.512140 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root
uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.512244 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:57.543752 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:57.544147 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:57.544468 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:57.544927 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:57.544966 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.545017 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:57.545045 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.548967 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:32789
I20240503 08:00:57.548993 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:32789 every 8 connection(s)
I20240503 08:00:57.549794 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:57.552350 3799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.553249 3802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.553814 3795 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:00:57.553900 3795 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:57.554153 3795 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
W20240503 08:00:57.554157 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.554877 3666 ts_manager.cc:194] Registered new tserver with Master: 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:00:57.554975 396 server_base.cc:1034] running on GCE node
I20240503 08:00:57.555162 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:57.555203 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:57.555219 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723257555219 us; error 0 us; skew 500 ppm
I20240503 08:00:57.555769 3666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48632
I20240503 08:00:57.556313 396 webserver.cc:460] Webserver started at http://127.0.99.2:33129/ using document root <none> and password file <none>
I20240503 08:00:57.556479 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:57.556533 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:57.556607 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:57.557058 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/instance:
uuid: "e01f09ad924f45c699fb12e45a597ac8"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.558486 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:00:57.559371 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.559624 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:57.559685 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root
uuid: "e01f09ad924f45c699fb12e45a597ac8"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.559742 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:57.582015 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:57.582392 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:57.582644 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:57.583096 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:57.583137 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.583171 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:57.583204 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.587338 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:43191
I20240503 08:00:57.587385 3870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:43191 every 8 connection(s)
I20240503 08:00:57.588075 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:57.590948 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:57.591665 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.592003 3871 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:00:57.592085 3871 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:57.592350 3871 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
W20240503 08:00:57.592608 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:57.593026 3666 ts_manager.cc:194] Registered new tserver with Master: e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:57.593542 396 server_base.cc:1034] running on GCE node
I20240503 08:00:57.593700 3666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48648
I20240503 08:00:57.593736 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:00:57.593770 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:00:57.593789 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723257593789 us; error 0 us; skew 500 ppm
I20240503 08:00:57.594833 396 webserver.cc:460] Webserver started at http://127.0.99.3:46607/ using document root <none> and password file <none>
I20240503 08:00:57.594957 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:57.595001 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:57.595057 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:57.595415 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/instance:
uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.596839 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:57.597697 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.597981 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:57.598044 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root
uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
format_stamp: "Formatted at 2024-05-03 08:00:57 on dist-test-slave-mxwh"
I20240503 08:00:57.598114 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:57.605437 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:57.605890 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:57.606176 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:57.606621 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:57.606694 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.606731 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:57.606761 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:57.611444 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:46363
I20240503 08:00:57.611480 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:46363 every 8 connection(s)
I20240503 08:00:57.616086 3946 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:00:57.616218 3946 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:57.616442 3946 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
I20240503 08:00:57.617007 3666 ts_manager.cc:194] Registered new tserver with Master: 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:57.617502 3666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48658
I20240503 08:00:57.617537 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005645664s
I20240503 08:00:58.556799 3795 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:00:58.594795 3871 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:00:58.618499 3946 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:00:58.643900 396 test_util.cc:268] Using random seed: 515388856
I20240503 08:00:58.652076 3666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48674:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:00:58.652741 3666 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:00:58.666643 3910 tablet_service.cc:1467] Processing CreateTablet for tablet c59c68fd5da24311a940047a49eb36ec (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:00:58.666769 3907 tablet_service.cc:1467] Processing CreateTablet for tablet 26e0121d8d964328ae012d07b34c26e7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:00:58.666736 3908 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:00:58.667035 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59c68fd5da24311a940047a49eb36ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.667121 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 5899a1db66d148be808923d5b15aa34f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:00:58.667366 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5899a1db66d148be808923d5b15aa34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.667529 3907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e0121d8d964328ae012d07b34c26e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.669195 3835 tablet_service.cc:1467] Processing CreateTablet for tablet c59c68fd5da24311a940047a49eb36ec (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:00:58.669291 3831 tablet_service.cc:1467] Processing CreateTablet for tablet 5899a1db66d148be808923d5b15aa34f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:00:58.669459 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59c68fd5da24311a940047a49eb36ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.669584 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.669739 3832 tablet_service.cc:1467] Processing CreateTablet for tablet 26e0121d8d964328ae012d07b34c26e7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:00:58.669948 3832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e0121d8d964328ae012d07b34c26e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.671530 3833 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:00:58.671801 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.672282 3967 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.672441 3834 tablet_service.cc:1467] Processing CreateTablet for tablet ade6e49ee7304d79b81c03cd034a815c (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:00:58.672502 3758 tablet_service.cc:1467] Processing CreateTablet for tablet ade6e49ee7304d79b81c03cd034a815c (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:00:58.672701 3758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade6e49ee7304d79b81c03cd034a815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.667125 3909 tablet_service.cc:1467] Processing CreateTablet for tablet ade6e49ee7304d79b81c03cd034a815c (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:00:58.672930 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade6e49ee7304d79b81c03cd034a815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.673000 3757 tablet_service.cc:1467] Processing CreateTablet for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:00:58.673228 3757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.673373 3967 tablet_bootstrap.cc:654] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.672716 3759 tablet_service.cc:1467] Processing CreateTablet for tablet c59c68fd5da24311a940047a49eb36ec (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:00:58.674224 3759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59c68fd5da24311a940047a49eb36ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.672701 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade6e49ee7304d79b81c03cd034a815c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.675334 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5899a1db66d148be808923d5b15aa34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.675726 3969 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.676415 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 26e0121d8d964328ae012d07b34c26e7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:00:58.676654 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e0121d8d964328ae012d07b34c26e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.676935 3969 tablet_bootstrap.cc:654] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.678128 3969 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.678233 3969 ts_tablet_manager.cc:1367] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:58.678611 3755 tablet_service.cc:1467] Processing CreateTablet for tablet 5899a1db66d148be808923d5b15aa34f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:00:58.678695 3969 raft_consensus.cc:348] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.678833 3969 raft_consensus.cc:374] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.678869 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5899a1db66d148be808923d5b15aa34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.678874 3969 raft_consensus.cc:729] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.679068 3969 consensus_queue.cc:260] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.679481 3969 ts_tablet_manager.cc:1398] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:58.679674 3969 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.680737 3969 tablet_bootstrap.cc:654] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.681126 3966 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.681217 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 81111e10c6c44e3182ff19e2001a824f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:00:58.681480 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81111e10c6c44e3182ff19e2001a824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.681627 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 81111e10c6c44e3182ff19e2001a824f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:00:58.681852 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81111e10c6c44e3182ff19e2001a824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.682080 3755 tablet_service.cc:1467] Processing CreateTablet for tablet 81111e10c6c44e3182ff19e2001a824f (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:00:58.682209 3966 tablet_bootstrap.cc:654] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.682328 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81111e10c6c44e3182ff19e2001a824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.683308 3967 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.683394 3967 ts_tablet_manager.cc:1367] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:00:58.683830 3967 raft_consensus.cc:348] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.683941 3967 raft_consensus.cc:374] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.683980 3967 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.684111 3967 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.684286 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 88348c5624c94fbcb125f112e65bdb34 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:00:58.684515 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.684566 3967 ts_tablet_manager.cc:1398] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:58.684733 3967 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.685211 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 9811bcfc2e9f495ca6115718712ac4ff (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:00:58.685614 3967 tablet_bootstrap.cc:654] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.686010 3910 tablet_service.cc:1467] Processing CreateTablet for tablet 20be03fba1ee443dba1709cbc69707bd (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:00:58.686259 3908 tablet_service.cc:1467] Processing CreateTablet for tablet 06177450ba6d45f48d6c3cf937737158 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:00:58.686656 3967 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.686713 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9811bcfc2e9f495ca6115718712ac4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.686798 3967 ts_tablet_manager.cc:1367] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:58.687146 3909 tablet_service.cc:1467] Processing CreateTablet for tablet 88348c5624c94fbcb125f112e65bdb34 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:00:58.687251 3967 raft_consensus.cc:348] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.687368 3967 raft_consensus.cc:374] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.687422 3967 raft_consensus.cc:729] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.687532 3967 consensus_queue.cc:260] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.687611 3757 tablet_service.cc:1467] Processing CreateTablet for tablet 06177450ba6d45f48d6c3cf937737158 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:00:58.687847 3757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06177450ba6d45f48d6c3cf937737158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.687901 3967 ts_tablet_manager.cc:1398] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.687999 3754 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbb6ccaa8d144c48404a95066a11ab1 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:00:58.688261 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbb6ccaa8d144c48404a95066a11ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.688128 3759 tablet_service.cc:1467] Processing CreateTablet for tablet 9811bcfc2e9f495ca6115718712ac4ff (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:00:58.688459 3759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9811bcfc2e9f495ca6115718712ac4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.688970 3834 tablet_service.cc:1467] Processing CreateTablet for tablet 9811bcfc2e9f495ca6115718712ac4ff (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:00:58.689119 3830 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbb6ccaa8d144c48404a95066a11ab1 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:00:58.689291 3832 tablet_service.cc:1467] Processing CreateTablet for tablet 20be03fba1ee443dba1709cbc69707bd (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:00:58.689880 3835 tablet_service.cc:1467] Processing CreateTablet for tablet 06177450ba6d45f48d6c3cf937737158 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:00:58.686617 3905 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbb6ccaa8d144c48404a95066a11ab1 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:00:58.686617 3907 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecce677bc4429bbab195d09e822bc7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:00:58.690992 3967 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.691098 3756 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecce677bc4429bbab195d09e822bc7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:00:58.687379 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.690992 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9811bcfc2e9f495ca6115718712ac4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.691423 3907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecce677bc4429bbab195d09e822bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.691466 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecce677bc4429bbab195d09e822bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.693374 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06177450ba6d45f48d6c3cf937737158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.693374 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06177450ba6d45f48d6c3cf937737158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.694607 3967 tablet_bootstrap.cc:654] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.695348 3830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbb6ccaa8d144c48404a95066a11ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.695509 3831 tablet_service.cc:1467] Processing CreateTablet for tablet a8ecce677bc4429bbab195d09e822bc7 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:00:58.695744 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecce677bc4429bbab195d09e822bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.697326 3967 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.697374 3905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbb6ccaa8d144c48404a95066a11ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.697422 3967 ts_tablet_manager.cc:1367] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:58.697759 3758 tablet_service.cc:1467] Processing CreateTablet for tablet 20be03fba1ee443dba1709cbc69707bd (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:00:58.697937 3967 raft_consensus.cc:348] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.698066 3967 raft_consensus.cc:374] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.698117 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20be03fba1ee443dba1709cbc69707bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.698017 3758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20be03fba1ee443dba1709cbc69707bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.698107 3967 raft_consensus.cc:729] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.699745 3967 consensus_queue.cc:260] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.700062 3967 ts_tablet_manager.cc:1398] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:58.700350 3832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20be03fba1ee443dba1709cbc69707bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.701064 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 88348c5624c94fbcb125f112e65bdb34 (DEFAULT_TABLE table=test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:00:58.701324 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:58.701462 3967 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.702225 3967 tablet_bootstrap.cc:654] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.703442 3969 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.703442 3967 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.703567 3969 ts_tablet_manager.cc:1367] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 08:00:58.703686 3966 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.703698 3967 ts_tablet_manager.cc:1367] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:58.704085 3969 raft_consensus.cc:348] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.704227 3969 raft_consensus.cc:374] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.704283 3969 raft_consensus.cc:729] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.704267 3967 raft_consensus.cc:348] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.704355 3967 raft_consensus.cc:374] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.704386 3967 raft_consensus.cc:729] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.703799 3966 ts_tablet_manager.cc:1367] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:00:58.704473 3967 consensus_queue.cc:260] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.704427 3969 consensus_queue.cc:260] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.704728 3969 ts_tablet_manager.cc:1398] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.704728 3967 ts_tablet_manager.cc:1398] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.704910 3969 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.704885 3966 raft_consensus.cc:348] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.704986 3966 raft_consensus.cc:374] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.705019 3966 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.705132 3966 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.705216 3967 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.705593 3966 ts_tablet_manager.cc:1398] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.705792 3966 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.705919 3969 tablet_bootstrap.cc:654] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.706060 3967 tablet_bootstrap.cc:654] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.706745 3966 tablet_bootstrap.cc:654] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.707183 3969 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.707212 3967 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.707267 3967 ts_tablet_manager.cc:1367] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:58.707267 3969 ts_tablet_manager.cc:1367] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:58.707738 3969 raft_consensus.cc:348] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.707861 3969 raft_consensus.cc:374] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.707898 3969 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.708025 3969 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.708299 3966 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.708351 3969 ts_tablet_manager.cc:1398] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.708376 3966 ts_tablet_manager.cc:1367] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:58.708511 3969 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.708750 3966 raft_consensus.cc:348] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.708853 3966 raft_consensus.cc:374] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.708899 3966 raft_consensus.cc:729] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.709041 3966 consensus_queue.cc:260] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.709317 3966 ts_tablet_manager.cc:1398] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.709398 3969 tablet_bootstrap.cc:654] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.709509 3966 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.710314 3966 tablet_bootstrap.cc:654] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.710558 3967 raft_consensus.cc:348] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.710672 3967 raft_consensus.cc:374] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.710717 3967 raft_consensus.cc:729] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.710827 3967 consensus_queue.cc:260] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.711100 3967 ts_tablet_manager.cc:1398] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 08:00:58.711253 3967 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.711455 3966 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.711539 3966 ts_tablet_manager.cc:1367] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:58.711872 3966 raft_consensus.cc:348] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.711951 3966 raft_consensus.cc:374] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.711985 3966 raft_consensus.cc:729] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.712121 3966 consensus_queue.cc:260] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.712240 3967 tablet_bootstrap.cc:654] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.712538 3966 ts_tablet_manager.cc:1398] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.712719 3966 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.712838 3969 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.712908 3969 ts_tablet_manager.cc:1367] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:58.713331 3969 raft_consensus.cc:348] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.713418 3969 raft_consensus.cc:374] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.713449 3969 raft_consensus.cc:729] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.713603 3966 tablet_bootstrap.cc:654] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.713552 3969 consensus_queue.cc:260] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.713822 3969 ts_tablet_manager.cc:1398] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.713999 3969 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.714599 3967 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.714694 3967 ts_tablet_manager.cc:1367] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:58.714829 3966 tablet_bootstrap.cc:492] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.714910 3966 ts_tablet_manager.cc:1367] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:58.714931 3969 tablet_bootstrap.cc:654] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.715035 3967 raft_consensus.cc:348] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.715106 3967 raft_consensus.cc:374] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.715126 3967 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.715220 3967 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.715349 3966 raft_consensus.cc:348] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.715464 3967 ts_tablet_manager.cc:1398] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:58.715462 3966 raft_consensus.cc:374] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.715557 3966 raft_consensus.cc:729] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.715651 3966 consensus_queue.cc:260] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.715890 3966 ts_tablet_manager.cc:1398] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.716061 3966 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.716478 3969 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.715687 3967 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.716681 3969 ts_tablet_manager.cc:1367] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:58.716982 3966 tablet_bootstrap.cc:654] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.717221 3969 raft_consensus.cc:348] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.717486 3967 tablet_bootstrap.cc:654] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.718505 3966 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.718583 3966 ts_tablet_manager.cc:1367] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:58.718139 3969 raft_consensus.cc:374] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.718912 3969 raft_consensus.cc:729] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.719009 3966 raft_consensus.cc:348] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.719106 3966 raft_consensus.cc:374] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.719148 3966 raft_consensus.cc:729] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.719269 3966 consensus_queue.cc:260] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.719574 3966 ts_tablet_manager.cc:1398] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.719734 3966 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.720053 3967 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.720140 3967 ts_tablet_manager.cc:1367] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:00:58.719269 3969 consensus_queue.cc:260] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.720679 3969 ts_tablet_manager.cc:1398] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:58.720687 3966 tablet_bootstrap.cc:654] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.721092 3969 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.721850 3966 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.721953 3966 ts_tablet_manager.cc:1367] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:58.722002 3969 tablet_bootstrap.cc:654] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.722934 3966 raft_consensus.cc:348] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.723042 3966 raft_consensus.cc:374] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.723083 3966 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.723171 3969 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.723210 3966 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.723377 3969 ts_tablet_manager.cc:1367] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:58.723513 3966 ts_tablet_manager.cc:1398] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:58.723691 3966 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.723996 3969 raft_consensus.cc:348] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.724092 3969 raft_consensus.cc:374] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.724130 3969 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.724304 3969 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.724668 3969 ts_tablet_manager.cc:1398] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.724793 3967 raft_consensus.cc:348] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.724881 3969 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.724895 3967 raft_consensus.cc:374] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.724973 3967 raft_consensus.cc:729] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.725075 3967 consensus_queue.cc:260] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.725346 3967 ts_tablet_manager.cc:1398] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:00:58.724668 3966 tablet_bootstrap.cc:654] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.725898 3969 tablet_bootstrap.cc:654] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.726615 3967 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.727630 3967 tablet_bootstrap.cc:654] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.728972 3966 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.729014 3969 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.729071 3966 ts_tablet_manager.cc:1367] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:00:58.729071 3969 ts_tablet_manager.cc:1367] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:00:58.729481 3966 raft_consensus.cc:348] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.729501 3969 raft_consensus.cc:348] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.729578 3966 raft_consensus.cc:374] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.729600 3969 raft_consensus.cc:374] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.729665 3969 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.729600 3966 raft_consensus.cc:729] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.729804 3969 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.729844 3966 consensus_queue.cc:260] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.730087 3969 ts_tablet_manager.cc:1398] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.730123 3966 ts_tablet_manager.cc:1398] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.730257 3969 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.730326 3966 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.731141 3969 tablet_bootstrap.cc:654] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.731220 3966 tablet_bootstrap.cc:654] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.732283 3969 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.732373 3969 ts_tablet_manager.cc:1367] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:58.732854 3969 raft_consensus.cc:348] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.732957 3967 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.732982 3969 raft_consensus.cc:374] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.733031 3967 ts_tablet_manager.cc:1367] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:00:58.733021 3969 raft_consensus.cc:729] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.733188 3969 consensus_queue.cc:260] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.733457 3969 ts_tablet_manager.cc:1398] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.733464 3967 raft_consensus.cc:348] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.733569 3967 raft_consensus.cc:374] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.733600 3967 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.733646 3969 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.733706 3967 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.733930 3967 ts_tablet_manager.cc:1398] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.734081 3967 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.735597 3967 tablet_bootstrap.cc:654] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.735738 3969 tablet_bootstrap.cc:654] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.736714 3967 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.736809 3967 ts_tablet_manager.cc:1367] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:58.736837 3969 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.736897 3969 ts_tablet_manager.cc:1367] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:58.737200 3967 raft_consensus.cc:348] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.737210 3969 raft_consensus.cc:348] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.737309 3967 raft_consensus.cc:374] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.737320 3969 raft_consensus.cc:374] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.737356 3967 raft_consensus.cc:729] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.737360 3969 raft_consensus.cc:729] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.737488 3967 consensus_queue.cc:260] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.737497 3969 consensus_queue.cc:260] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.737742 3967 ts_tablet_manager.cc:1398] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.737763 3969 ts_tablet_manager.cc:1398] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.737932 3969 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.737962 3967 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.738786 3967 tablet_bootstrap.cc:654] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.738925 3969 tablet_bootstrap.cc:654] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.739925 3967 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.740007 3967 ts_tablet_manager.cc:1367] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:58.740478 3967 raft_consensus.cc:348] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.740587 3967 raft_consensus.cc:374] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.740630 3967 raft_consensus.cc:729] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.740744 3967 consensus_queue.cc:260] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.741008 3967 ts_tablet_manager.cc:1398] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.741143 3967 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.741887 3967 tablet_bootstrap.cc:654] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.742794 3966 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.742884 3966 ts_tablet_manager.cc:1367] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 08:00:58.742955 3967 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.743077 3967 ts_tablet_manager.cc:1367] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:58.743294 3966 raft_consensus.cc:348] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.743391 3966 raft_consensus.cc:374] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.743412 3966 raft_consensus.cc:729] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.743450 3967 raft_consensus.cc:348] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.743531 3966 consensus_queue.cc:260] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.743557 3967 raft_consensus.cc:374] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.743644 3967 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.743762 3967 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.743876 3966 ts_tablet_manager.cc:1398] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:58.744040 3967 ts_tablet_manager.cc:1398] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:58.744051 3966 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.744300 3967 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Bootstrap starting.
I20240503 08:00:58.745222 3966 tablet_bootstrap.cc:654] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.745280 3967 tablet_bootstrap.cc:654] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.746443 3969 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.746496 3966 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.746536 3969 ts_tablet_manager.cc:1367] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:58.746582 3966 ts_tablet_manager.cc:1367] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:58.746975 3969 raft_consensus.cc:348] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.747089 3969 raft_consensus.cc:374] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.747123 3969 raft_consensus.cc:729] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.747057 3966 raft_consensus.cc:348] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.747186 3966 raft_consensus.cc:374] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.747225 3966 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.747242 3969 consensus_queue.cc:260] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.747349 3966 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.747534 3969 ts_tablet_manager.cc:1398] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:58.747623 3966 ts_tablet_manager.cc:1398] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.747730 3969 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.747821 3966 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.748795 3966 tablet_bootstrap.cc:654] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.748805 3969 tablet_bootstrap.cc:654] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.749949 3966 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.750038 3966 ts_tablet_manager.cc:1367] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:58.750471 3966 raft_consensus.cc:348] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.750586 3966 raft_consensus.cc:374] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.750638 3966 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.750761 3966 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.751075 3966 ts_tablet_manager.cc:1398] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:58.751260 3966 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.752014 3966 tablet_bootstrap.cc:654] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.753078 3967 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: No bootstrap required, opened a new log
I20240503 08:00:58.753141 3969 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.753177 3967 ts_tablet_manager.cc:1367] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:58.753209 3969 ts_tablet_manager.cc:1367] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:00:58.753461 3966 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.753561 3966 ts_tablet_manager.cc:1367] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:58.753612 3967 raft_consensus.cc:348] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.753715 3967 raft_consensus.cc:374] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.753755 3967 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.753896 3967 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.753935 3966 raft_consensus.cc:348] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.754019 3966 raft_consensus.cc:374] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.754048 3966 raft_consensus.cc:729] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.754174 3967 ts_tablet_manager.cc:1398] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.754173 3966 consensus_queue.cc:260] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.754429 3969 raft_consensus.cc:348] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.754523 3969 raft_consensus.cc:374] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.754565 3969 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.754654 3966 ts_tablet_manager.cc:1398] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.754680 3969 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.754845 3966 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: Bootstrap starting.
I20240503 08:00:58.754945 3969 ts_tablet_manager.cc:1398] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:58.755406 3969 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Bootstrap starting.
I20240503 08:00:58.755781 3966 tablet_bootstrap.cc:654] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.756377 3969 tablet_bootstrap.cc:654] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:58.757509 3966 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: No bootstrap required, opened a new log
I20240503 08:00:58.757597 3966 ts_tablet_manager.cc:1367] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:58.757932 3966 raft_consensus.cc:348] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.758031 3966 raft_consensus.cc:374] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.758070 3966 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.758203 3966 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.758318 3969 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: No bootstrap required, opened a new log
I20240503 08:00:58.758396 3969 ts_tablet_manager.cc:1367] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:58.758477 3966 ts_tablet_manager.cc:1398] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:58.758822 3969 raft_consensus.cc:348] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.758917 3969 raft_consensus.cc:374] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:58.758960 3969 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Initialized, Role: FOLLOWER
I20240503 08:00:58.759076 3969 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.759689 3969 ts_tablet_manager.cc:1398] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:58.780674 3971 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.780834 3971 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.781401 3971 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.784741 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.784752 3845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.784902 3920 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.784937 3845 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.785194 3734 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.785342 3971 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.785398 3971 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.785439 3971 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.786232 3971 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.786588 3971 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.786896 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.786971 3845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.787007 3920 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.787070 3845 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.787815 3920 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.787871 3845 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.788107 3733 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.788285 3971 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.788487 3971 raft_consensus.cc:686] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.788616 3971 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:58.790205 3666 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:58.802584 3979 raft_consensus.cc:482] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.802726 3979 raft_consensus.cc:504] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.803177 3979 leader_election.cc:288] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.807579 3845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.807588 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" is_pre_election: true
I20240503 08:00:58.807755 3769 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.807740 3845 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.808104 3884 leader_election.cc:302] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.808315 3979 raft_consensus.cc:2781] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.808377 3979 raft_consensus.cc:482] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.808409 3979 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.809280 3979 raft_consensus.cc:504] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.809585 3979 leader_election.cc:288] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.809973 3845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.810010 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:58.810074 3845 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.810128 3769 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.810230 3971 raft_consensus.cc:482] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.810305 3971 raft_consensus.cc:504] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.810667 3971 leader_election.cc:288] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.811033 3845 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.811210 3769 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.811272 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.811384 3844 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.811618 3733 leader_election.cc:302] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.811668 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.811668 3884 leader_election.cc:302] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.811806 3920 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.811829 3971 raft_consensus.cc:2781] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.811900 3971 raft_consensus.cc:482] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.811964 3971 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.812036 3979 raft_consensus.cc:2781] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.812276 3979 raft_consensus.cc:686] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:00:58.812420 3979 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.812827 3971 raft_consensus.cc:504] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.813159 3971 leader_election.cc:288] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.813410 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.813410 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.813529 3844 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.813557 3920 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.813967 3666 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:58.814209 3844 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.814395 3920 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.814440 3733 leader_election.cc:302] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.815074 3971 raft_consensus.cc:2781] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.815205 3971 raft_consensus.cc:686] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.815323 3971 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.816751 3666 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.818919 3971 raft_consensus.cc:482] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.819005 3971 raft_consensus.cc:504] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.819311 3971 leader_election.cc:288] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.819648 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.819648 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.819808 3920 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.819815 3844 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.820063 3733 leader_election.cc:302] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.820256 3971 raft_consensus.cc:2781] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.820330 3971 raft_consensus.cc:482] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.820387 3971 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.821172 3971 raft_consensus.cc:504] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.821532 3971 leader_election.cc:288] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.821779 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.821776 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.821880 3920 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.821969 3844 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.822616 3844 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.822614 3920 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.822885 3734 leader_election.cc:302] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.823030 3971 raft_consensus.cc:2781] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.823200 3971 raft_consensus.cc:686] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.823349 3971 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.824867 3666 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.830224 3971 raft_consensus.cc:482] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.830332 3971 raft_consensus.cc:504] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.830687 3971 leader_election.cc:288] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.831063 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6e49ee7304d79b81c03cd034a815c" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.831058 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6e49ee7304d79b81c03cd034a815c" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.831187 3844 raft_consensus.cc:2454] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.831204 3920 raft_consensus.cc:2454] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.831449 3733 leader_election.cc:302] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.831660 3971 raft_consensus.cc:2781] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.831717 3971 raft_consensus.cc:482] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.831764 3971 raft_consensus.cc:3037] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.832566 3971 raft_consensus.cc:504] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.832862 3971 leader_election.cc:288] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.833108 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6e49ee7304d79b81c03cd034a815c" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.833155 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6e49ee7304d79b81c03cd034a815c" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.833210 3844 raft_consensus.cc:3037] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.833236 3920 raft_consensus.cc:3037] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.833885 3844 raft_consensus.cc:2454] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.833932 3920 raft_consensus.cc:2454] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.834136 3733 leader_election.cc:302] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.834331 3971 raft_consensus.cc:2781] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.834430 3971 raft_consensus.cc:686] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.834553 3971 consensus_queue.cc:237] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.835505 3974 raft_consensus.cc:482] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.835595 3974 raft_consensus.cc:504] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.836037 3974 leader_election.cc:288] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.836221 3666 catalog_manager.cc:5484] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.839839 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" is_pre_election: true
I20240503 08:00:58.839960 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.839970 3769 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 0.
I20240503 08:00:58.840092 3920 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 0.
I20240503 08:00:58.840288 3809 leader_election.cc:302] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.840500 3974 raft_consensus.cc:2781] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.840548 3974 raft_consensus.cc:482] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.840603 3974 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.841408 3974 raft_consensus.cc:504] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.841727 3974 leader_election.cc:288] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.842057 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.842172 3920 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.842384 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:58.842478 3769 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.842882 3920 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 1.
I20240503 08:00:58.843133 3810 leader_election.cc:302] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.843154 3769 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 1.
I20240503 08:00:58.843286 3974 raft_consensus.cc:2781] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.843365 3974 raft_consensus.cc:686] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:00:58.843509 3974 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.845194 3666 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.853572 3971 raft_consensus.cc:482] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.853708 3971 raft_consensus.cc:504] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.854094 3971 leader_election.cc:288] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.854457 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.854493 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.854568 3844 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.854640 3920 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.854817 3733 leader_election.cc:302] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.854979 3971 raft_consensus.cc:2781] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.855031 3971 raft_consensus.cc:482] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.855069 3971 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.855852 3971 raft_consensus.cc:504] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.856187 3971 leader_election.cc:288] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.856559 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.856679 3844 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.857404 3844 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.857594 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.857695 3920 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.857669 3733 leader_election.cc:302] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.857843 3971 raft_consensus.cc:2781] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.857928 3971 raft_consensus.cc:686] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.858063 3971 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.858456 3920 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.859766 3666 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.870025 3976 raft_consensus.cc:482] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.870182 3976 raft_consensus.cc:504] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.870610 3976 leader_election.cc:288] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.871064 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.871064 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" is_pre_election: true
I20240503 08:00:58.871232 3769 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 0.
I20240503 08:00:58.871232 3920 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 0.
I20240503 08:00:58.871522 3810 leader_election.cc:302] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.871683 3976 raft_consensus.cc:2781] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.871743 3976 raft_consensus.cc:482] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.871793 3976 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.872699 3976 raft_consensus.cc:504] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.873041 3976 leader_election.cc:288] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.873473 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.873473 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:58.873629 3920 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.873694 3769 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.874495 3920 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 1.
I20240503 08:00:58.874497 3769 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 1.
I20240503 08:00:58.874815 3810 leader_election.cc:302] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.874974 3976 raft_consensus.cc:2781] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.875041 3976 raft_consensus.cc:686] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:00:58.875173 3976 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.876947 3666 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.892241 3979 raft_consensus.cc:482] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.892396 3979 raft_consensus.cc:504] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.892825 3979 leader_election.cc:288] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.893373 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.893373 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" is_pre_election: true
I20240503 08:00:58.893574 3769 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.893574 3844 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.893893 3884 leader_election.cc:302] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.894078 3979 raft_consensus.cc:2781] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.894140 3979 raft_consensus.cc:482] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.894196 3979 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.895066 3979 raft_consensus.cc:504] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.895385 3979 leader_election.cc:288] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.895773 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:58.895790 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.895918 3769 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.895918 3844 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.896857 3769 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.896857 3844 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.897183 3884 leader_election.cc:302] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.897329 3979 raft_consensus.cc:2781] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.897416 3979 raft_consensus.cc:686] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:00:58.897553 3979 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.899200 3666 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:58.944304 3979 raft_consensus.cc:482] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.944495 3979 raft_consensus.cc:504] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.944914 3979 leader_election.cc:288] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.945386 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" is_pre_election: true
I20240503 08:00:58.945394 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.945515 3769 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.945549 3844 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 0.
I20240503 08:00:58.945765 3884 leader_election.cc:302] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.945942 3979 raft_consensus.cc:2781] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.946002 3979 raft_consensus.cc:482] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.946039 3979 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.946837 3979 raft_consensus.cc:504] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.947146 3979 leader_election.cc:288] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:00:58.947499 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.947480 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:58.947629 3844 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.947677 3769 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.948514 3844 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.948514 3769 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 1.
I20240503 08:00:58.948887 3884 leader_election.cc:302] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.949121 3979 raft_consensus.cc:2781] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.949203 3979 raft_consensus.cc:686] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:00:58.949333 3979 consensus_queue.cc:237] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.951144 3666 catalog_manager.cc:5484] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:58.955631 4011 raft_consensus.cc:482] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.955776 4011 raft_consensus.cc:504] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.956118 4011 leader_election.cc:288] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.956557 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.956557 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.956732 3920 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.956791 3844 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.956993 3734 leader_election.cc:302] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.957201 4011 raft_consensus.cc:2781] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.957252 4011 raft_consensus.cc:482] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.957278 4011 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.958061 4011 raft_consensus.cc:504] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.958366 4011 leader_election.cc:288] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.958703 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.958801 3920 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.959756 3920 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.959968 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.960018 3734 leader_election.cc:302] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.960076 3844 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.960258 4011 raft_consensus.cc:2781] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.960350 4011 raft_consensus.cc:686] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.960490 4011 consensus_queue.cc:237] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.961006 3844 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.962154 3666 catalog_manager.cc:5484] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:58.980672 3971 raft_consensus.cc:482] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:58.980823 3971 raft_consensus.cc:504] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.981266 3971 leader_election.cc:288] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.981681 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" is_pre_election: true
I20240503 08:00:58.981685 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8" is_pre_election: true
I20240503 08:00:58.981793 3920 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.981863 3844 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 0.
I20240503 08:00:58.982098 3734 leader_election.cc:302] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:00:58.982264 3971 raft_consensus.cc:2781] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:58.982316 3971 raft_consensus.cc:482] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:58.982342 3971 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.983184 3971 raft_consensus.cc:504] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.983520 3971 leader_election.cc:288] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:58.983937 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:58.983937 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:58.984097 3844 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.984097 3920 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:58.985073 3844 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.985073 3920 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 1.
I20240503 08:00:58.985488 3733 leader_election.cc:302] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:58.985666 3971 raft_consensus.cc:2781] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:58.985747 3971 raft_consensus.cc:686] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:58.985882 3971 consensus_queue.cc:237] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:58.987457 3666 catalog_manager.cc:5484] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:59.066269 3844 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.067382 4015 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.068181 3769 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.068542 3768 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.071579 3920 raft_consensus.cc:1261] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.072027 3920 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.072221 3971 consensus_queue.cc:1035] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.072844 3971 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.073644 3845 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.074368 3845 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.074510 4015 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.074540 3979 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.074949 3844 raft_consensus.cc:1261] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.075480 3979 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.075755 3979 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.076272 3768 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.076967 3971 consensus_queue.cc:1035] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.077011 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.077355 3843 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.078281 4011 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.079200 3918 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.079699 3976 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.080886 4015 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.081346 3916 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.081526 4011 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.082010 3845 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.082245 3918 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.083575 3974 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.084287 3971 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.086405 3766 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.086943 3765 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.087662 3840 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.087961 3839 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.088250 3845 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.089085 4011 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.092074 3914 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.092300 4072 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.092398 3913 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.092372 3974 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.092479 3976 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.092527 4069 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.092490 4071 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.095758 4011 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.096047 4011 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.098516 3919 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.103710 3840 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:59.104966 3971 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.106513 3971 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:59.113305 4045 mvcc.cc:205] Tried to move back new op lower bound from 7023506469147467776 to 7023506468445995008. Current Snapshot: MvccSnapshot[applied={T|T < 7023506469133946880}]
I20240503 08:00:59.125941 4041 mvcc.cc:205] Tried to move back new op lower bound from 7023506469233692672 to 7023506468142043136. Current Snapshot: MvccSnapshot[applied={T|T < 7023506469155987456}]
I20240503 08:00:59.144450 4096 mvcc.cc:205] Tried to move back new op lower bound from 7023506469166616576 to 7023506467999617024. Current Snapshot: MvccSnapshot[applied={T|T < 7023506469166616576}]
I20240503 08:00:59.469691 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:00:59.512460 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:00:59.512622 3767 raft_consensus.cc:595] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS e01f09ad924f45c699fb12e45a597ac8
I20240503 08:00:59.513746 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:00:59.513857 3767 raft_consensus.cc:595] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS e01f09ad924f45c699fb12e45a597ac8
I20240503 08:00:59.517843 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:00:59.517978 3767 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS 68a5c03f0bc8469c9a6f1b31546a3ba3
I20240503 08:00:59.522634 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:00:59.522778 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 08:00:59.529413 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.530835 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.531333 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.546912 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.547468 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.548007 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.548547 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.550086 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.551731 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.553611 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.558590 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.558717 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.562285 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.562383 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.563167 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575079 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575136 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575136 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575680 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575716 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.575780 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.576241 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.576321 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.576354 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.576375 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.576802 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.580235 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.581017 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.581694 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.582429 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.582545 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.583482 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.583554 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.583673 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.583673 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.584044 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.584044 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.584779 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.585567 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.585623 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.586341 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.588335 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.588450 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.588450 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.590425 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.590456 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.591296 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.591339 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.592145 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.592249 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.592265 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.592473 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.593322 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.595281 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.597309 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.597397 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.597525 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.597397 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.597983 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.600003 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.599957 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.599957 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.600941 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.601802 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.603766 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.604591 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.604588 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.604724 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.606818 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.606925 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.606925 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.610116 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.610162 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.610237 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.610986 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.611053 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.614061 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.614100 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.615068 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.615109 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.615120 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.617347 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.617434 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.617527 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.620532 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.620618 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.621392 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.621455 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.623294 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.625216 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.625285 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.627276 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.627396 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.627396 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.629441 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.629441 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.630264 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.632421 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.632469 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.632539 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.633365 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.635298 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.637199 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.637259 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.637980 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.638700 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.642848 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.642894 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.642848 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.643836 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.645749 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.645901 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.646663 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.646831 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.650954 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.650964 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.651760 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.652563 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.655640 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.655656 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.656428 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.657167 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.657923 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.659984 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.660025 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.660866 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.663873 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.665915 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.665895 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.666844 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.666878 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.667732 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.670770 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.670807 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.672863 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.672863 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.674860 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.676755 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.678684 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.680601 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.681406 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.681566 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.682266 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.683054 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.684964 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.686916 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.688951 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.688958 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.689708 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.691589 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.694566 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.695376 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.696239 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.696365 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.698349 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.700520 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.700564 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.703743 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.704558 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.705243 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.708240 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.709021 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.710965 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.711068 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.712889 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.713712 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.713757 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.715734 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.716604 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.721827 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.721891 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.721891 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.723932 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.727010 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.729053 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.729053 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.730051 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.730074 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.732064 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.732775 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.733417 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.737324 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.741420 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.741420 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.742231 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.742897 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.747094 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.747125 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.747205 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.749076 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.751173 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.751955 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.752745 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.756791 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.759773 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.760598 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.760672 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.761436 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.766633 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.766849 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.766855 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.767452 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.770437 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.772387 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.772387 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.778635 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.778801 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.778808 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.781909 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.781924 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.786012 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.786073 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.787977 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.788741 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.791682 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.795843 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.795840 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.797923 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.798041 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.798041 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.801050 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.802888 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.805851 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.807782 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.807782 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.810801 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.814790 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.818909 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.818944 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.819104 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.819746 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.821595 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.823503 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.824265 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.827178 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.827842 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.828543 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.833420 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.837374 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.840463 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.840592 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.842610 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.842701 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.846817 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.846904 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.847671 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.848403 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.852406 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.854400 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.857403 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.862552 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.862604 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.865713 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.866580 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.868638 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.871625 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.872501 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.872501 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.874486 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.876384 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.877197 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.882306 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.886479 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.886551 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.889598 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.890492 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.893438 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.895313 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.895364 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.900449 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.900569 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.900579 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.901307 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.908344 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.913542 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.913739 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.914232 3971 raft_consensus.cc:979] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:00:59.914330 3971 raft_consensus.cc:1067] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
W20240503 08:00:59.914759 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.914705 3918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
from {username='slave'} at 127.0.0.1:58522
I20240503 08:00:59.914865 3918 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:59.914945 3918 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:59.915846 3918 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:59.916180 3918 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 2 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:00:59.916486 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:00:59.916507 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:59.916663 3767 raft_consensus.cc:3032] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:59.916682 3844 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:59.916709 3767 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:59.916860 3767 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
I20240503 08:00:59.917006 3767 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:59.917544 3844 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 2.
W20240503 08:00:59.917840 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.918015 3767 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 2.
I20240503 08:00:59.918208 3884 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:59.918377 4109 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:59.918471 4109 raft_consensus.cc:686] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:00:59.918605 4109 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } }
W20240503 08:00:59.919762 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.920038 3662 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:59.920519 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.921248 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.926371 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.927242 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.929173 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.929293 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.933372 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.940820 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.941668 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.943663 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.945578 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.945722 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.947576 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.949517 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.954056 3844 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:00:59.954072 3767 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:00:59.954424 4106 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:00:59.954576 4109 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20240503 08:00:59.962287 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.968575 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.969048 4069 raft_consensus.cc:979] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:00:59.969125 4069 raft_consensus.cc:1067] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:00:59.969499 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:45694
I20240503 08:00:59.969619 3844 raft_consensus.cc:482] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:59.969687 3844 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:00:59.970606 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.970606 3844 raft_consensus.cc:504] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:59.970993 3844 leader_election.cc:288] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:00:59.971246 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:00:59.971252 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:00:59.971354 3916 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:59.971383 3767 raft_consensus.cc:3032] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:59.971431 3767 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:00:59.971570 3767 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:00:59.971688 3767 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:59.972258 3916 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:00:59.972513 3767 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:00:59.972596 3810 leader_election.cc:302] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:00:59.972769 4059 raft_consensus.cc:2781] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:59.972859 4059 raft_consensus.cc:686] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:00:59.972986 4059 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
W20240503 08:00:59.972592 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.974051 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:00:59.974380 3662 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:59.974551 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.977836 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.980070 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.990671 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:00:59.997308 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:00:59.999514 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:01:00.000711 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.003540 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.004925 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:01:00.007889 3916 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:01:00.008332 3767 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:01:00.008400 4059 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:01:00.011004 3974 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20240503 08:01:00.019078 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.029757 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.033510 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.036371 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
W20240503 08:01:00.049054 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Service unavailable: leader transfer in progress
I20240503 08:01:00.052232 3971 raft_consensus.cc:979] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:00.052353 3971 raft_consensus.cc:1067] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:00.052739 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:45694
I20240503 08:01:00.052868 3844 raft_consensus.cc:482] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:00.052945 3844 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:00.053833 3844 raft_consensus.cc:504] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:00.054188 3844 leader_election.cc:288] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:00.054500 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:00.054500 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:00.054661 3767 raft_consensus.cc:3032] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:00.054675 3916 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:00.054718 3767 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:00.054912 3767 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:00.055080 3767 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:00.055765 3916 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:01:00.055970 3767 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:01:00.056102 3810 leader_election.cc:302] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:00.056326 4108 raft_consensus.cc:2781] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:00.056422 4108 raft_consensus.cc:686] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:00.056562 4108 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:00.058161 3662 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
W20240503 08:01:00.059240 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:01:00.067708 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:01:00.069980 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
W20240503 08:01:00.082306 3749 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34970: Illegal state: replica 44f0a2c3ee394d70ad7145e6b04ad926 is not leader of this config: current role FOLLOWER
I20240503 08:01:00.091924 3767 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:01:00.091944 3916 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:01:00.092334 3976 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:01:00.092483 4108 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:01:00.116741 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:01:00.119282 4113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:01:00.120292 4114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:01:00.121191 4116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:01:00.122041 396 server_base.cc:1034] running on GCE node
I20240503 08:01:00.122220 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:01:00.122257 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:01:00.122273 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723260122273 us; error 0 us; skew 500 ppm
I20240503 08:01:00.123383 396 webserver.cc:460] Webserver started at http://127.0.99.4:45445/ using document root <none> and password file <none>
I20240503 08:01:00.123522 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:01:00.123569 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:01:00.123626 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:01:00.124065 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-3-root/instance:
uuid: "be55fde3594e417faf23e55a23944033"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.125622 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:01:00.126574 4121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.126931 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:01:00.127007 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-3-root
uuid: "be55fde3594e417faf23e55a23944033"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.127094 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:01:00.143282 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:01:00.143728 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:01:00.144099 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:01:00.144580 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:01:00.144620 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.144670 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:01:00.144704 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.149503 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:34743
I20240503 08:01:00.149550 4184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:34743 every 8 connection(s)
I20240503 08:01:00.150314 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:01:00.152879 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:01:00.153846 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:01:00.154155 4185 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:01:00.154244 4185 heartbeater.cc:464] Registering TS with master...
I20240503 08:01:00.154448 4185 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
W20240503 08:01:00.154794 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:01:00.155120 3659 ts_manager.cc:194] Registered new tserver with Master: be55fde3594e417faf23e55a23944033 (127.0.99.4:34743)
I20240503 08:01:00.155696 396 server_base.cc:1034] running on GCE node
I20240503 08:01:00.155725 3659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48682
I20240503 08:01:00.155886 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:01:00.155930 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:01:00.155956 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723260155954 us; error 0 us; skew 500 ppm
I20240503 08:01:00.157071 396 webserver.cc:460] Webserver started at http://127.0.99.5:34643/ using document root <none> and password file <none>
I20240503 08:01:00.157212 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:01:00.157265 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:01:00.157343 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:01:00.157753 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-4-root/instance:
uuid: "f8e707d0d0eb4812b9e22be35d86a152"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.159518 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:01:00.160552 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.160775 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:01:00.160859 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-4-root
uuid: "f8e707d0d0eb4812b9e22be35d86a152"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.160957 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:01:00.170281 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:01:00.170661 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:01:00.170961 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:01:00.171402 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:01:00.171442 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.171491 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:01:00.171519 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.176041 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.5:37779
I20240503 08:01:00.176119 4260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.5:37779 every 8 connection(s)
I20240503 08:01:00.176880 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:01:00.179353 4264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:01:00.180298 4265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:01:00.180837 4261 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:01:00.180939 4261 heartbeater.cc:464] Registering TS with master...
I20240503 08:01:00.181144 4261 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
W20240503 08:01:00.181223 4267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:01:00.181710 3659 ts_manager.cc:194] Registered new tserver with Master: f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779)
I20240503 08:01:00.182050 396 server_base.cc:1034] running on GCE node
I20240503 08:01:00.182228 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:01:00.182274 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:01:00.182303 396 hybrid_clock.cc:648] HybridClock initialized: now 1714723260182303 us; error 0 us; skew 500 ppm
I20240503 08:01:00.182346 3659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48690
I20240503 08:01:00.183423 396 webserver.cc:460] Webserver started at http://127.0.99.6:41063/ using document root <none> and password file <none>
I20240503 08:01:00.183547 396 fs_manager.cc:359] Metadata directory not provided
I20240503 08:01:00.183593 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:01:00.183648 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:01:00.183990 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-5-root/instance:
uuid: "a08972a8b584403aaf1a7fb51b775a79"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.185357 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:01:00.186194 4272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.186419 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:01:00.186478 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-5-root
uuid: "a08972a8b584403aaf1a7fb51b775a79"
format_stamp: "Formatted at 2024-05-03 08:01:00 on dist-test-slave-mxwh"
I20240503 08:01:00.186542 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:01:00.197294 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:01:00.197707 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:01:00.198199 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:01:00.198629 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:01:00.198668 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.198725 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:01:00.198755 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:01:00.203514 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.6:43249
I20240503 08:01:00.203603 4335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.6:43249 every 8 connection(s)
I20240503 08:01:00.208238 4336 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44179
I20240503 08:01:00.208334 4336 heartbeater.cc:464] Registering TS with master...
I20240503 08:01:00.208537 4336 heartbeater.cc:511] Master 127.0.99.62:44179 requested a full tablet report, sending...
I20240503 08:01:00.209087 3659 ts_manager.cc:194] Registered new tserver with Master: a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249)
I20240503 08:01:00.209607 3659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48694
I20240503 08:01:00.480058 3916 consensus_queue.cc:237] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:00.481554 3767 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:01:00.481626 3844 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:01:00.481937 4341 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:00.482236 4106 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:00.483633 4342 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:00.484052 3767 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:00.484453 3844 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:00.486821 3659 catalog_manager.cc:5484] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
W20240503 08:01:00.488987 3886 consensus_peers.cc:485] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5899a1db66d148be808923d5b15aa34f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.490741 3842 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:00.492000 3767 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:00.492000 3916 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:00.492507 3974 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:00.492686 4108 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:00.494171 3974 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:00.494442 3916 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
W20240503 08:01:00.496119 3811 consensus_peers.cc:485] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81111e10c6c44e3182ff19e2001a824f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.496563 3767 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:00.497174 3662 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:00.503098 3767 consensus_queue.cc:237] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:00.504307 3842 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.504338 3916 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.504643 3971 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:00.504842 4349 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:00.506399 3971 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.506865 3842 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.507889 3916 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.508689 3657 catalog_manager.cc:5484] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:00.508751 3767 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:00.510775 3916 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.511075 4349 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 08:01:00.511905 3732 consensus_peers.cc:485] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6b5bb3a0ab4bbc9314b9cd3634d373. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.512555 3842 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.513075 4353 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:00.513013 3971 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:00.514145 3916 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
W20240503 08:01:00.516345 3735 consensus_peers.cc:485] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88348c5624c94fbcb125f112e65bdb34. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.516311 3659 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:00.516580 3844 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:00.517203 3842 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:00.518662 3916 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:00.519248 4059 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:00.520432 3767 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:00.520731 4059 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:00.521013 4108 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:00.522753 3767 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
W20240503 08:01:00.523202 3811 consensus_peers.cc:485] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06177450ba6d45f48d6c3cf937737158. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.523259 3659 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index -1 to 29, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:00.523752 3842 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:00.524412 3916 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:00.525969 3767 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.526063 3916 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:00.526404 3974 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:00.526715 4108 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:00.528267 3976 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.528663 3767 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.528685 3916 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:00.530813 3662 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index -1 to 28, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
W20240503 08:01:00.533605 3808 consensus_peers.cc:485] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9811bcfc2e9f495ca6115718712ac4ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:00.921449 4354 ts_tablet_manager.cc:902] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:00.922169 4354 tablet_copy_client.cc:308] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:00.925626 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet 81111e10c6c44e3182ff19e2001a824f from peer f8e707d0d0eb4812b9e22be35d86a152 ({username='slave'} at 127.0.0.1:45740)
I20240503 08:01:00.925747 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet 81111e10c6c44e3182ff19e2001a824f from peer f8e707d0d0eb4812b9e22be35d86a152 at {username='slave'} at 127.0.0.1:45740: session id = f8e707d0d0eb4812b9e22be35d86a152-81111e10c6c44e3182ff19e2001a824f
I20240503 08:01:00.926939 3855 tablet_copy_source_session.cc:205] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:00.927767 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81111e10c6c44e3182ff19e2001a824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:00.929837 4354 tablet_copy_client.cc:783] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:00.930004 4354 tablet_copy_client.cc:647] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:00.930943 4358 tablet_copy_client.cc:1002] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:00.938268 4360 ts_tablet_manager.cc:902] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:00.938855 4360 tablet_copy_client.cc:308] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:32789
I20240503 08:01:00.942378 3779 tablet_copy_service.cc:140] P 44f0a2c3ee394d70ad7145e6b04ad926: Received BeginTabletCopySession request for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:35004)
I20240503 08:01:00.942492 3779 tablet_copy_service.cc:161] P 44f0a2c3ee394d70ad7145e6b04ad926: Beginning new tablet copy session on tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:35004: session id = be55fde3594e417faf23e55a23944033-cf6b5bb3a0ab4bbc9314b9cd3634d373
I20240503 08:01:00.943307 3779 tablet_copy_source_session.cc:205] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:00.944214 4360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:00.946337 4360 tablet_copy_client.cc:783] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:00.946521 4360 tablet_copy_client.cc:647] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:00.947492 4364 tablet_copy_client.cc:1002] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:00.971134 4366 ts_tablet_manager.cc:902] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:00.971706 4366 tablet_copy_client.cc:308] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:32789
I20240503 08:01:00.975365 3779 tablet_copy_service.cc:140] P 44f0a2c3ee394d70ad7145e6b04ad926: Received BeginTabletCopySession request for tablet 88348c5624c94fbcb125f112e65bdb34 from peer f8e707d0d0eb4812b9e22be35d86a152 ({username='slave'} at 127.0.0.1:35020)
I20240503 08:01:00.975483 3779 tablet_copy_service.cc:161] P 44f0a2c3ee394d70ad7145e6b04ad926: Beginning new tablet copy session on tablet 88348c5624c94fbcb125f112e65bdb34 from peer f8e707d0d0eb4812b9e22be35d86a152 at {username='slave'} at 127.0.0.1:35020: session id = f8e707d0d0eb4812b9e22be35d86a152-88348c5624c94fbcb125f112e65bdb34
I20240503 08:01:00.976532 3779 tablet_copy_source_session.cc:205] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:00.977348 4366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:00.979168 4371 ts_tablet_manager.cc:902] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:00.979617 4366 tablet_copy_client.cc:783] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:00.979799 4366 tablet_copy_client.cc:647] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:00.979945 4371 tablet_copy_client.cc:308] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:46363
I20240503 08:01:00.981034 4370 tablet_copy_client.cc:1002] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:00.984004 3930 tablet_copy_service.cc:140] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Received BeginTabletCopySession request for tablet 5899a1db66d148be808923d5b15aa34f from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:58570)
I20240503 08:01:00.984097 3930 tablet_copy_service.cc:161] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Beginning new tablet copy session on tablet 5899a1db66d148be808923d5b15aa34f from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:58570: session id = a08972a8b584403aaf1a7fb51b775a79-5899a1db66d148be808923d5b15aa34f
I20240503 08:01:00.984997 3930 tablet_copy_source_session.cc:205] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:00.985733 4371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5899a1db66d148be808923d5b15aa34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:00.987689 4371 tablet_copy_client.cc:783] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:00.987825 4371 tablet_copy_client.cc:647] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:00.988837 4375 tablet_copy_client.cc:1002] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:00.998623 4378 ts_tablet_manager.cc:902] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:00.999253 4378 tablet_copy_client.cc:308] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:01.003299 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet 06177450ba6d45f48d6c3cf937737158 from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:45746)
I20240503 08:01:01.003408 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet 06177450ba6d45f48d6c3cf937737158 from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:45746: session id = a08972a8b584403aaf1a7fb51b775a79-06177450ba6d45f48d6c3cf937737158
I20240503 08:01:01.004243 3855 tablet_copy_source_session.cc:205] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:01.004951 4378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06177450ba6d45f48d6c3cf937737158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:01.006695 4378 tablet_copy_client.cc:783] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:01.006862 4378 tablet_copy_client.cc:647] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:01.008196 4382 tablet_copy_client.cc:1002] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:01.070223 4383 ts_tablet_manager.cc:902] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:01.070766 4383 tablet_copy_client.cc:308] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:01.074493 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet 9811bcfc2e9f495ca6115718712ac4ff from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:45752)
I20240503 08:01:01.074610 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet 9811bcfc2e9f495ca6115718712ac4ff from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:45752: session id = be55fde3594e417faf23e55a23944033-9811bcfc2e9f495ca6115718712ac4ff
I20240503 08:01:01.075572 3855 tablet_copy_source_session.cc:205] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:01.076345 4383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9811bcfc2e9f495ca6115718712ac4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:01.078424 4383 tablet_copy_client.cc:783] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:01.078594 4383 tablet_copy_client.cc:647] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:01.079553 4387 tablet_copy_client.cc:1002] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:01.156703 4185 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:01:01.183195 4261 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:01:01.210348 4336 heartbeater.cc:503] Master 127.0.99.62:44179 was elected leader, sending a full tablet report...
I20240503 08:01:01.523025 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:01:01.524223 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:01:01.524338 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:01:02.931219 4354 tablet_copy_client.cc:523] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:02.932874 4354 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap starting.
I20240503 08:01:02.947777 4360 tablet_copy_client.cc:523] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:02.949740 4360 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Bootstrap starting.
I20240503 08:01:02.958671 4354 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:02.958986 4354 tablet_bootstrap.cc:492] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap complete.
I20240503 08:01:02.959177 4354 ts_tablet_manager.cc:1367] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.001s
I20240503 08:01:02.959625 4354 raft_consensus.cc:348] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:02.959772 4354 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f8e707d0d0eb4812b9e22be35d86a152, State: Initialized, Role: LEARNER
I20240503 08:01:02.959885 4354 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:02.960245 4354 ts_tablet_manager.cc:1398] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:01:02.960904 3855 tablet_copy_service.cc:342] P e01f09ad924f45c699fb12e45a597ac8: Request end of tablet copy session f8e707d0d0eb4812b9e22be35d86a152-81111e10c6c44e3182ff19e2001a824f received from {username='slave'} at 127.0.0.1:45740
I20240503 08:01:02.960984 3855 tablet_copy_service.cc:433] P e01f09ad924f45c699fb12e45a597ac8: ending tablet copy session f8e707d0d0eb4812b9e22be35d86a152-81111e10c6c44e3182ff19e2001a824f on tablet 81111e10c6c44e3182ff19e2001a824f with peer f8e707d0d0eb4812b9e22be35d86a152
I20240503 08:01:02.976629 4360 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:02.976939 4360 tablet_bootstrap.cc:492] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Bootstrap complete.
I20240503 08:01:02.977142 4360 ts_tablet_manager.cc:1367] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.003s
I20240503 08:01:02.977612 4360 raft_consensus.cc:348] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:02.977738 4360 raft_consensus.cc:729] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55fde3594e417faf23e55a23944033, State: Initialized, Role: LEARNER
I20240503 08:01:02.977856 4360 consensus_queue.cc:260] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:02.978147 4360 ts_tablet_manager.cc:1398] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:01:02.978577 3779 tablet_copy_service.cc:342] P 44f0a2c3ee394d70ad7145e6b04ad926: Request end of tablet copy session be55fde3594e417faf23e55a23944033-cf6b5bb3a0ab4bbc9314b9cd3634d373 received from {username='slave'} at 127.0.0.1:35004
I20240503 08:01:02.978654 3779 tablet_copy_service.cc:433] P 44f0a2c3ee394d70ad7145e6b04ad926: ending tablet copy session be55fde3594e417faf23e55a23944033-cf6b5bb3a0ab4bbc9314b9cd3634d373 on tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 with peer be55fde3594e417faf23e55a23944033
I20240503 08:01:02.981310 4366 tablet_copy_client.cc:523] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:02.982614 4366 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap starting.
I20240503 08:01:02.989107 4371 tablet_copy_client.cc:523] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:02.990188 4371 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Bootstrap starting.
I20240503 08:01:03.005185 4366 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:03.005538 4366 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap complete.
I20240503 08:01:03.005724 4366 ts_tablet_manager.cc:1367] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240503 08:01:03.006173 4366 raft_consensus.cc:348] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:03.006315 4366 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8e707d0d0eb4812b9e22be35d86a152, State: Initialized, Role: LEARNER
I20240503 08:01:03.006415 4366 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:03.006722 4366 ts_tablet_manager.cc:1398] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:01:03.007169 3779 tablet_copy_service.cc:342] P 44f0a2c3ee394d70ad7145e6b04ad926: Request end of tablet copy session f8e707d0d0eb4812b9e22be35d86a152-88348c5624c94fbcb125f112e65bdb34 received from {username='slave'} at 127.0.0.1:35020
I20240503 08:01:03.007252 3779 tablet_copy_service.cc:433] P 44f0a2c3ee394d70ad7145e6b04ad926: ending tablet copy session f8e707d0d0eb4812b9e22be35d86a152-88348c5624c94fbcb125f112e65bdb34 on tablet 88348c5624c94fbcb125f112e65bdb34 with peer f8e707d0d0eb4812b9e22be35d86a152
I20240503 08:01:03.008447 4378 tablet_copy_client.cc:523] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:03.009912 4378 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap starting.
I20240503 08:01:03.012615 4371 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:03.012904 4371 tablet_bootstrap.cc:492] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Bootstrap complete.
I20240503 08:01:03.013092 4371 ts_tablet_manager.cc:1367] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.005s
I20240503 08:01:03.013547 4371 raft_consensus.cc:348] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:03.013686 4371 raft_consensus.cc:729] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Initialized, Role: LEARNER
I20240503 08:01:03.013813 4371 consensus_queue.cc:260] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:03.014158 4371 ts_tablet_manager.cc:1398] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:01:03.014598 3930 tablet_copy_service.cc:342] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Request end of tablet copy session a08972a8b584403aaf1a7fb51b775a79-5899a1db66d148be808923d5b15aa34f received from {username='slave'} at 127.0.0.1:58570
I20240503 08:01:03.014675 3930 tablet_copy_service.cc:433] P 68a5c03f0bc8469c9a6f1b31546a3ba3: ending tablet copy session a08972a8b584403aaf1a7fb51b775a79-5899a1db66d148be808923d5b15aa34f on tablet 5899a1db66d148be808923d5b15aa34f with peer a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:03.031289 4378 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:03.031628 4378 tablet_bootstrap.cc:492] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap complete.
I20240503 08:01:03.031841 4378 ts_tablet_manager.cc:1367] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20240503 08:01:03.031900 4159 raft_consensus.cc:1203] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 08:01:03.032352 4378 raft_consensus.cc:348] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:03.032492 4378 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Initialized, Role: LEARNER
I20240503 08:01:03.032603 4378 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:03.032874 4378 ts_tablet_manager.cc:1398] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:01:03.033322 3855 tablet_copy_service.cc:342] P e01f09ad924f45c699fb12e45a597ac8: Request end of tablet copy session a08972a8b584403aaf1a7fb51b775a79-06177450ba6d45f48d6c3cf937737158 received from {username='slave'} at 127.0.0.1:45746
I20240503 08:01:03.033402 3855 tablet_copy_service.cc:433] P e01f09ad924f45c699fb12e45a597ac8: ending tablet copy session a08972a8b584403aaf1a7fb51b775a79-06177450ba6d45f48d6c3cf937737158 on tablet 06177450ba6d45f48d6c3cf937737158 with peer a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:03.079864 4383 tablet_copy_client.cc:523] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:03.081182 4383 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Bootstrap starting.
I20240503 08:01:03.103994 4383 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:03.104351 4383 tablet_bootstrap.cc:492] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Bootstrap complete.
I20240503 08:01:03.104563 4383 ts_tablet_manager.cc:1367] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.003s
I20240503 08:01:03.105063 4383 raft_consensus.cc:348] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:03.105197 4383 raft_consensus.cc:729] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55fde3594e417faf23e55a23944033, State: Initialized, Role: LEARNER
I20240503 08:01:03.105310 4383 consensus_queue.cc:260] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:03.105597 4383 ts_tablet_manager.cc:1398] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:01:03.106142 3855 tablet_copy_service.cc:342] P e01f09ad924f45c699fb12e45a597ac8: Request end of tablet copy session be55fde3594e417faf23e55a23944033-9811bcfc2e9f495ca6115718712ac4ff received from {username='slave'} at 127.0.0.1:45752
I20240503 08:01:03.106218 3855 tablet_copy_service.cc:433] P e01f09ad924f45c699fb12e45a597ac8: ending tablet copy session be55fde3594e417faf23e55a23944033-9811bcfc2e9f495ca6115718712ac4ff on tablet 9811bcfc2e9f495ca6115718712ac4ff with peer be55fde3594e417faf23e55a23944033
I20240503 08:01:03.145166 4310 raft_consensus.cc:1203] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 08:01:03.203508 4235 raft_consensus.cc:1203] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 08:01:03.363667 4310 raft_consensus.cc:1203] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 08:01:03.436213 4069 raft_consensus.cc:1050] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: attempting to promote NON_VOTER be55fde3594e417faf23e55a23944033 to VOTER
I20240503 08:01:03.436611 4069 consensus_queue.cc:237] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:03.437989 4159 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.437999 3842 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.438418 4404 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.438469 3916 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.438771 4402 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.438961 4404 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.441275 4401 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.441709 4159 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.441895 3918 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.443694 3842 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.444114 3659 catalog_manager.cc:5484] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.446810 4159 raft_consensus.cc:1203] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 08:01:03.449527 3767 consensus_queue.cc:237] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:03.450534 4159 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:03.450548 3844 raft_consensus.cc:1261] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:03.450892 4401 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.451189 4401 consensus_queue.cc:1035] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.452275 4401 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.452832 3842 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.453269 4159 raft_consensus.cc:2932] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:03.454217 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:03.455011 3657 catalog_manager.cc:5484] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index 29 to 30, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:03.457543 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 with delete_type TABLET_DATA_TOMBSTONED (TS 68a5c03f0bc8469c9a6f1b31546a3ba3 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:58516
I20240503 08:01:03.457942 4412 tablet_replica.cc:330] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: stopping tablet replica
I20240503 08:01:03.458076 4412 raft_consensus.cc:2229] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:03.458240 4412 raft_consensus.cc:2258] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:03.458957 4412 ts_tablet_manager.cc:1875] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:03.461118 4412 ts_tablet_manager.cc:1888] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 08:01:03.461200 4412 log.cc:1196] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/wals/cf6b5bb3a0ab4bbc9314b9cd3634d373
I20240503 08:01:03.461748 3652 catalog_manager.cc:4830] TS 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363): tablet cf6b5bb3a0ab4bbc9314b9cd3634d373 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:03.500835 4235 raft_consensus.cc:1203] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 08:01:03.524544 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:01:03.534960 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:45758
I20240503 08:01:03.535122 3842 raft_consensus.cc:595] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:03.535928 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:01:03.536057 3767 raft_consensus.cc:595] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS e01f09ad924f45c699fb12e45a597ac8
I20240503 08:01:03.536410 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 08:01:03.536530 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:01:03.548457 4069 raft_consensus.cc:979] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:03.548591 4069 raft_consensus.cc:1067] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:03.548961 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:45694
I20240503 08:01:03.549086 3844 raft_consensus.cc:482] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:03.549156 3844 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:03.549940 3844 raft_consensus.cc:504] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:03.550244 3844 leader_election.cc:288] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:03.550547 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:03.550662 3767 raft_consensus.cc:3032] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:03.550717 3767 raft_consensus.cc:729] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:03.550763 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ecce677bc4429bbab195d09e822bc7" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:03.550890 3918 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:03.550860 3767 consensus_queue.cc:260] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:03.551082 3767 raft_consensus.cc:3037] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:03.551913 3767 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:01:03.551913 3918 raft_consensus.cc:2454] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 2.
I20240503 08:01:03.552266 3810 leader_election.cc:302] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:03.552450 4059 raft_consensus.cc:2781] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:03.552623 4059 raft_consensus.cc:686] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:03.552752 4059 consensus_queue.cc:237] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } }
I20240503 08:01:03.554409 3662 catalog_manager.cc:5484] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:03.563863 4417 raft_consensus.cc:1050] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: attempting to promote NON_VOTER a08972a8b584403aaf1a7fb51b775a79 to VOTER
I20240503 08:01:03.564395 4417 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 23, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:03.565837 3767 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:03.565896 3918 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:03.566200 4419 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.566318 4310 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEARNER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:03.566588 4059 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.566773 4419 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.568444 4059 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.569061 4310 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.570276 3918 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.570766 3767 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.571166 3662 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 29 to 30, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:03.574244 3844 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:03.575569 3767 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:01:03.575569 4310 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:01:03.575927 4415 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:03.576068 4417 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:03.577780 4417 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.578240 3767 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.580026 4310 raft_consensus.cc:2932] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.580549 3661 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 30 to 31, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:03.580695 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 06177450ba6d45f48d6c3cf937737158 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:03.582749 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet 06177450ba6d45f48d6c3cf937737158 with delete_type TABLET_DATA_TOMBSTONED (TS 68a5c03f0bc8469c9a6f1b31546a3ba3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:58516
I20240503 08:01:03.582976 4412 tablet_replica.cc:330] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: stopping tablet replica
I20240503 08:01:03.583118 4412 raft_consensus.cc:2229] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:03.583242 4412 raft_consensus.cc:2258] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:03.583920 4412 ts_tablet_manager.cc:1875] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:03.585803 4412 ts_tablet_manager.cc:1888] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 08:01:03.585875 4412 log.cc:1196] T 06177450ba6d45f48d6c3cf937737158 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/wals/06177450ba6d45f48d6c3cf937737158
I20240503 08:01:03.586357 3652 catalog_manager.cc:4830] TS 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363): tablet 06177450ba6d45f48d6c3cf937737158 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:03.713618 4415 raft_consensus.cc:979] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:03.713765 4415 raft_consensus.cc:1067] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:03.714108 3766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34962
I20240503 08:01:03.714227 3766 raft_consensus.cc:482] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:03.714285 3766 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:03.715164 3766 raft_consensus.cc:504] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:03.715526 3766 leader_election.cc:288] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:03.715760 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:03.715864 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:03.715883 3844 raft_consensus.cc:3032] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:01:03.715955 3844 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:03.715965 3918 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:03.716109 3844 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:03.716370 3844 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:03.716969 3918 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:03.717217 3734 leader_election.cc:302] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:01:03.717308 3844 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:03.717370 4069 raft_consensus.cc:2781] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:01:03.717456 4069 raft_consensus.cc:686] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:03.717571 4069 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:03.719259 3657 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 2 to 3, leader changed from e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:03.724072 3766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:03.724210 3766 raft_consensus.cc:595] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
I20240503 08:01:03.727308 3766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:03.727406 3766 raft_consensus.cc:595] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
W20240503 08:01:03.727660 3651 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 81111e10c6c44e3182ff19e2001a824f already in progress [suppressed 8 similar messages]
I20240503 08:01:03.728257 3766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:03.728344 3766 raft_consensus.cc:595] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
W20240503 08:01:03.728689 3714 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 81111e10c6c44e3182ff19e2001a824f already in progress
W20240503 08:01:03.728745 3714 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 81111e10c6c44e3182ff19e2001a824f already in progress
I20240503 08:01:03.824560 4425 raft_consensus.cc:1050] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: attempting to promote NON_VOTER a08972a8b584403aaf1a7fb51b775a79 to VOTER
I20240503 08:01:03.824995 4425 consensus_queue.cc:237] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:03.826313 4310 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:03.826473 3844 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:03.826642 4425 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:03.826821 3766 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:03.826944 4427 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:03.827394 4391 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:03.829052 4428 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.829447 3844 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.829480 3767 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.831297 4310 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.832115 3657 catalog_manager.cc:5484] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.836025 3918 consensus_queue.cc:237] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:03.837041 4310 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.837191 3767 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.837342 4428 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.837707 4428 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.838773 4427 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.839229 4310 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.839530 3767 raft_consensus.cc:2932] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.841238 3661 catalog_manager.cc:5484] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 reported cstate change: config changed from index 28 to 29, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:03.841459 3652 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5899a1db66d148be808923d5b15aa34f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:03.843364 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 5899a1db66d148be808923d5b15aa34f with delete_type TABLET_DATA_TOMBSTONED (TS e01f09ad924f45c699fb12e45a597ac8 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:03.843897 4432 tablet_replica.cc:330] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: stopping tablet replica
I20240503 08:01:03.844082 4432 raft_consensus.cc:2229] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:03.844269 4432 raft_consensus.cc:2258] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:03.844908 4432 ts_tablet_manager.cc:1875] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:03.847043 4432 ts_tablet_manager.cc:1888] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 08:01:03.847106 4432 log.cc:1196] T 5899a1db66d148be808923d5b15aa34f P e01f09ad924f45c699fb12e45a597ac8: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/wals/5899a1db66d148be808923d5b15aa34f
I20240503 08:01:03.847514 3651 catalog_manager.cc:4830] TS e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191): tablet 5899a1db66d148be808923d5b15aa34f (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:03.886288 4401 raft_consensus.cc:1050] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
I20240503 08:01:03.886770 4401 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:03.888290 3918 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.888432 4235 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 LEARNER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.888482 3844 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:03.888761 4401 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.888918 4069 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.889799 4401 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:03.891095 4069 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.891553 4235 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.892446 3842 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.893985 3662 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.894989 3918 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.898555 3766 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:03.899662 3918 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:03.899770 4235 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:03.899986 4404 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.900149 4069 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:03.901526 4404 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.901947 4235 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.902927 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88348c5624c94fbcb125f112e65bdb34 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:03.903705 3657 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.904409 3918 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:03.906085 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 88348c5624c94fbcb125f112e65bdb34 with delete_type TABLET_DATA_TOMBSTONED (TS e01f09ad924f45c699fb12e45a597ac8 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:03.906291 4432 tablet_replica.cc:330] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: stopping tablet replica
I20240503 08:01:03.906428 4432 raft_consensus.cc:2229] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:03.906549 4432 raft_consensus.cc:2258] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:03.907145 4432 ts_tablet_manager.cc:1875] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:03.909114 4432 ts_tablet_manager.cc:1888] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 08:01:03.909201 4432 log.cc:1196] T 88348c5624c94fbcb125f112e65bdb34 P e01f09ad924f45c699fb12e45a597ac8: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/wals/88348c5624c94fbcb125f112e65bdb34
I20240503 08:01:03.909638 3651 catalog_manager.cc:4830] TS e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191): tablet 88348c5624c94fbcb125f112e65bdb34 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:03.940129 3766 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:01:03.940596 4415 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:03.943993 3918 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:01:03.945255 4415 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:03.998302 4419 raft_consensus.cc:1050] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: attempting to promote NON_VOTER be55fde3594e417faf23e55a23944033 to VOTER
I20240503 08:01:03.998800 4419 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:04.000144 3766 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:04.000226 4159 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:04.000551 3918 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:04.000620 4059 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:04.000828 4417 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:04.000990 4415 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:04.003086 4417 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.003525 3766 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.004547 3918 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.005790 4159 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.006147 3659 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 28 to 29, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:04.009236 3844 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:04.010290 3918 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:04.010290 4159 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:04.010663 4059 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.010957 4415 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.011999 4059 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.012801 3918 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.014060 4159 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:04.014303 3662 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:04.015405 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9811bcfc2e9f495ca6115718712ac4ff with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:04.016837 3755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9811bcfc2e9f495ca6115718712ac4ff with delete_type TABLET_DATA_TOMBSTONED (TS 44f0a2c3ee394d70ad7145e6b04ad926 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:04.017261 4447 tablet_replica.cc:330] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: stopping tablet replica
I20240503 08:01:04.017419 4447 raft_consensus.cc:2229] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:04.017560 4447 raft_consensus.cc:2258] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:04.018092 4447 ts_tablet_manager.cc:1875] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:04.019775 4447 ts_tablet_manager.cc:1888] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 08:01:04.019843 4447 log.cc:1196] T 9811bcfc2e9f495ca6115718712ac4ff P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root/wals/9811bcfc2e9f495ca6115718712ac4ff
I20240503 08:01:04.020351 3651 catalog_manager.cc:4830] TS 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789): tablet 9811bcfc2e9f495ca6115718712ac4ff (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:04.276033 4235 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Advancing to term 3
I20240503 08:01:04.277063 4235 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 3 LEARNER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:01:04.277583 4069 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.279743 4401 raft_consensus.cc:1050] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
W20240503 08:01:04.279827 4401 raft_consensus.cc:1053] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Unable to promote non-voter f8e707d0d0eb4812b9e22be35d86a152: Service unavailable: leader transfer in progress
I20240503 08:01:04.312960 3918 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:01:04.313444 4401 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.315059 4069 raft_consensus.cc:979] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:04.315163 4069 raft_consensus.cc:1067] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Signalling peer 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:04.316282 3844 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:01:04.316327 3916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
from {username='slave'} at 127.0.0.1:58522
I20240503 08:01:04.316464 3916 raft_consensus.cc:482] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:04.316524 3916 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:04.316978 4401 raft_consensus.cc:1050] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
W20240503 08:01:04.317050 4401 raft_consensus.cc:1053] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Unable to promote non-voter f8e707d0d0eb4812b9e22be35d86a152: Service unavailable: leader transfer in progress
I20240503 08:01:04.317183 4401 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.317665 3916 raft_consensus.cc:504] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:04.317857 3918 raft_consensus.cc:1226] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Rejecting Update request from peer 44f0a2c3ee394d70ad7145e6b04ad926 for earlier term 3. Current term is 4. Ops: []
I20240503 08:01:04.318065 3916 leader_election.cc:288] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:04.318365 4401 consensus_queue.cc:1046] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: INVALID_TERM, Last received: 3.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:04.318511 4401 raft_consensus.cc:3032] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:01:04.318564 4401 raft_consensus.cc:729] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:04.318698 4401 consensus_queue.cc:260] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:04.318773 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:04.318862 4401 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:04.319996 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81111e10c6c44e3182ff19e2001a824f" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:04.320744 3766 raft_consensus.cc:2454] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 4.
I20240503 08:01:04.321025 3884 leader_election.cc:302] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters: e01f09ad924f45c699fb12e45a597ac8
I20240503 08:01:04.321174 4428 raft_consensus.cc:2781] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:01:04.321261 4428 raft_consensus.cc:686] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:04.321386 4428 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:04.322748 3657 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 3 to 4, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:04.787350 4235 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 3 LEARNER]: Advancing to term 4
I20240503 08:01:04.788336 4235 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 4 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:04.788785 4428 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:04.790292 4428 raft_consensus.cc:1050] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
W20240503 08:01:04.790365 4428 raft_consensus.cc:1053] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: Unable to promote non-voter f8e707d0d0eb4812b9e22be35d86a152: Illegal state: Leader has not yet committed an operation in its own term
I20240503 08:01:04.823746 3766 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:04.824187 4428 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:04.826601 3844 raft_consensus.cc:3037] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:04.827847 3844 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:04.828505 4457 raft_consensus.cc:1050] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
I20240503 08:01:04.828555 4428 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:04.828986 4457 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:04.830111 4235 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 4 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:01:04.830224 3844 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:01:04.830366 3766 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:01:04.830709 4456 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:04.830899 4428 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:04.830997 4457 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:04.832298 4428 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 29 to 32, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.832656 3766 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.833464 3844 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.834247 4235 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.835305 3657 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index 29 to 32, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.838850 3916 consensus_queue.cc:237] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:04.839769 3844 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 08:01:04.839769 4235 raft_consensus.cc:1261] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 08:01:04.840101 4425 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:04.840288 4456 consensus_queue.cc:1035] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:04.841724 4457 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.842157 3844 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.842259 4235 raft_consensus.cc:2932] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.843844 3652 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81111e10c6c44e3182ff19e2001a824f with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:04.844245 3662 catalog_manager.cc:5484] T 81111e10c6c44e3182ff19e2001a824f P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 32 to 33, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:04.846419 3755 tablet_service.cc:1514] Processing DeleteTablet for tablet 81111e10c6c44e3182ff19e2001a824f with delete_type TABLET_DATA_TOMBSTONED (TS 44f0a2c3ee394d70ad7145e6b04ad926 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:04.846848 4465 tablet_replica.cc:330] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: stopping tablet replica
I20240503 08:01:04.846983 4465 raft_consensus.cc:2229] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:04.847115 4465 raft_consensus.cc:2258] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:04.847609 4465 ts_tablet_manager.cc:1875] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:04.849507 4465 ts_tablet_manager.cc:1888] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20240503 08:01:04.849576 4465 log.cc:1196] T 81111e10c6c44e3182ff19e2001a824f P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root/wals/81111e10c6c44e3182ff19e2001a824f
I20240503 08:01:04.850013 3651 catalog_manager.cc:4830] TS 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789): tablet 81111e10c6c44e3182ff19e2001a824f (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:05.019513 3916 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:05.020992 3766 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:05.021010 3844 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:05.021415 4428 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:05.021616 4456 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 08:01:05.021832 3886 consensus_peers.cc:485] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c59c68fd5da24311a940047a49eb36ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.023233 4425 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.023674 3766 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.023699 3844 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.026113 3659 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index -1 to 29, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.027590 3916 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.029404 3844 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:01:05.029747 4456 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:05.029892 3766 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:01:05.030301 4456 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:05.031868 4457 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
W20240503 08:01:05.033418 3883 consensus_peers.cc:485] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20be03fba1ee443dba1709cbc69707bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.033818 3844 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.034253 3662 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index -1 to 27, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.034970 3766 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.036576 3842 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:05.037953 3766 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 08:01:05.038216 3811 consensus_peers.cc:485] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dbb6ccaa8d144c48404a95066a11ab1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.038355 3916 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:01:05.038405 4452 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:05.039057 4452 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:01:05.040793 4415 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.041143 3766 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.041620 3916 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.043483 3662 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index -1 to 27, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.044293 3916 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:05.045832 3842 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:05.046068 3766 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:05.046202 4425 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 08:01:05.046463 3886 consensus_peers.cc:485] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26e0121d8d964328ae012d07b34c26e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.046483 4456 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:05.048076 4425 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.048470 3842 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.048532 3766 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.050573 3659 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index -1 to 28, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:05.051129 3842 consensus_queue.cc:237] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:05.052587 3766 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240503 08:01:05.053150 3811 consensus_peers.cc:485] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8ecce677bc4429bbab195d09e822bc7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.053222 4452 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:05.053613 3916 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:01:05.054121 4452 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:05.054894 4059 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.055310 3766 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.057430 3766 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.057693 3662 catalog_manager.cc:5484] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.057665 3916 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:05.059171 3918 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240503 08:01:05.059408 3732 consensus_peers.cc:485] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88348c5624c94fbcb125f112e65bdb34. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:05.059528 4235 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 08:01:05.059698 4483 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:05.062012 4483 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:05.062716 4459 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.063180 3916 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.064179 4235 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.066249 3661 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 30 to 31, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:05.448277 4486 ts_tablet_manager.cc:902] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:05.448832 4486 tablet_copy_client.cc:308] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:32789
I20240503 08:01:05.449257 3779 tablet_copy_service.cc:140] P 44f0a2c3ee394d70ad7145e6b04ad926: Received BeginTabletCopySession request for tablet 88348c5624c94fbcb125f112e65bdb34 from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:35004)
I20240503 08:01:05.449355 3779 tablet_copy_service.cc:161] P 44f0a2c3ee394d70ad7145e6b04ad926: Beginning new tablet copy session on tablet 88348c5624c94fbcb125f112e65bdb34 from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:35004: session id = be55fde3594e417faf23e55a23944033-88348c5624c94fbcb125f112e65bdb34
I20240503 08:01:05.450277 3779 tablet_copy_source_session.cc:205] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:05.451037 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:05.453744 4486 tablet_copy_client.cc:783] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:05.454007 4486 tablet_copy_client.cc:647] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:05.455062 4490 tablet_copy_client.cc:1002] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:05.495656 4491 ts_tablet_manager.cc:902] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.496310 4491 tablet_copy_client.cc:308] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:46363
I20240503 08:01:05.496665 3930 tablet_copy_service.cc:140] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Received BeginTabletCopySession request for tablet 26e0121d8d964328ae012d07b34c26e7 from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:58570)
I20240503 08:01:05.496770 3930 tablet_copy_service.cc:161] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Beginning new tablet copy session on tablet 26e0121d8d964328ae012d07b34c26e7 from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:58570: session id = a08972a8b584403aaf1a7fb51b775a79-26e0121d8d964328ae012d07b34c26e7
I20240503 08:01:05.497789 3930 tablet_copy_source_session.cc:205] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:05.498543 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e0121d8d964328ae012d07b34c26e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:05.500707 4491 tablet_copy_client.cc:783] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:05.500900 4491 tablet_copy_client.cc:647] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:05.501926 4495 tablet_copy_client.cc:1002] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:05.535398 4496 ts_tablet_manager.cc:902] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Initiating tablet copy from peer 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.536048 4496 tablet_copy_client.cc:308] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:46363
I20240503 08:01:05.536777 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:01:05.538558 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:45758
I20240503 08:01:05.538690 3842 raft_consensus.cc:595] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:05.540474 3930 tablet_copy_service.cc:140] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Received BeginTabletCopySession request for tablet c59c68fd5da24311a940047a49eb36ec from peer f8e707d0d0eb4812b9e22be35d86a152 ({username='slave'} at 127.0.0.1:51120)
I20240503 08:01:05.540557 3930 tablet_copy_service.cc:161] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Beginning new tablet copy session on tablet c59c68fd5da24311a940047a49eb36ec from peer f8e707d0d0eb4812b9e22be35d86a152 at {username='slave'} at 127.0.0.1:51120: session id = f8e707d0d0eb4812b9e22be35d86a152-c59c68fd5da24311a940047a49eb36ec
I20240503 08:01:05.541612 3930 tablet_copy_source_session.cc:205] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:05.542412 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59c68fd5da24311a940047a49eb36ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:05.543819 3918 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:51122
I20240503 08:01:05.543978 3918 raft_consensus.cc:595] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:05.544633 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
mode: GRACEFUL
new_leader_uuid: "a08972a8b584403aaf1a7fb51b775a79"
from {username='slave'} at 127.0.0.1:51122
I20240503 08:01:05.544735 3916 raft_consensus.cc:595] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Received request to transfer leadership to TS a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:05.544880 4496 tablet_copy_client.cc:783] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:05.545042 4496 tablet_copy_client.cc:647] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:05.545553 3766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "f8e707d0d0eb4812b9e22be35d86a152"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:01:05.545655 3766 raft_consensus.cc:595] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS f8e707d0d0eb4812b9e22be35d86a152
I20240503 08:01:05.545941 4499 tablet_copy_client.cc:1002] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:05.546298 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
new_leader_uuid: "be55fde3594e417faf23e55a23944033"
from {username='slave'} at 127.0.0.1:45758
I20240503 08:01:05.546397 3842 raft_consensus.cc:595] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Received request to transfer leadership to TS be55fde3594e417faf23e55a23944033
I20240503 08:01:05.547058 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:51122
I20240503 08:01:05.547160 3916 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:05.547435 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 08:01:05.547569 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:01:05.571008 4505 ts_tablet_manager.cc:902] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:05.571517 4505 tablet_copy_client.cc:308] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:05.572011 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet a8ecce677bc4429bbab195d09e822bc7 from peer f8e707d0d0eb4812b9e22be35d86a152 ({username='slave'} at 127.0.0.1:45740)
I20240503 08:01:05.572110 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet a8ecce677bc4429bbab195d09e822bc7 from peer f8e707d0d0eb4812b9e22be35d86a152 at {username='slave'} at 127.0.0.1:45740: session id = f8e707d0d0eb4812b9e22be35d86a152-a8ecce677bc4429bbab195d09e822bc7
I20240503 08:01:05.573122 3855 tablet_copy_source_session.cc:205] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:05.573901 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ecce677bc4429bbab195d09e822bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:05.575929 4505 tablet_copy_client.cc:783] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:05.576089 4505 tablet_copy_client.cc:647] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:05.577108 4508 tablet_copy_client.cc:1002] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:05.580148 4483 raft_consensus.cc:979] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower f8e707d0d0eb4812b9e22be35d86a152 to start an election
I20240503 08:01:05.580269 4483 raft_consensus.cc:1067] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer f8e707d0d0eb4812b9e22be35d86a152 to start an election
I20240503 08:01:05.580648 4235 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34"
dest_uuid: "f8e707d0d0eb4812b9e22be35d86a152"
from {username='slave'} at 127.0.0.1:53544
I20240503 08:01:05.580806 4235 raft_consensus.cc:482] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.580873 4235 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.581768 4235 raft_consensus.cc:504] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.582299 4235 leader_election.cc:288] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.582531 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "f8e707d0d0eb4812b9e22be35d86a152" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:05.582644 3766 raft_consensus.cc:3032] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:05.582703 3766 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:05.582726 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88348c5624c94fbcb125f112e65bdb34" candidate_uuid: "f8e707d0d0eb4812b9e22be35d86a152" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:05.582825 3918 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.582852 3766 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.583063 3766 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.583547 3918 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e707d0d0eb4812b9e22be35d86a152 in term 2.
I20240503 08:01:05.583833 3766 raft_consensus.cc:2454] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e707d0d0eb4812b9e22be35d86a152 in term 2.
I20240503 08:01:05.583834 4200 leader_election.cc:302] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, f8e707d0d0eb4812b9e22be35d86a152; no voters:
I20240503 08:01:05.584239 4511 raft_consensus.cc:2781] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:05.584442 4511 raft_consensus.cc:686] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEADER]: Becoming Leader. State: Replica: f8e707d0d0eb4812b9e22be35d86a152, State: Running, Role: LEADER
I20240503 08:01:05.584607 4511 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.586314 3662 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5). New cstate: current_term: 2 leader_uuid: "f8e707d0d0eb4812b9e22be35d86a152" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.588189 4456 raft_consensus.cc:979] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.588287 4456 raft_consensus.cc:1067] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.588686 3766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34952
I20240503 08:01:05.588794 3766 raft_consensus.cc:482] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.588856 3766 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.589663 3766 raft_consensus.cc:504] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.589993 3766 leader_election.cc:288] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 2 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.590299 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:05.590411 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:05.590517 3918 raft_consensus.cc:3032] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:05.590560 3918 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:05.590698 3918 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.590884 3918 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.591804 3918 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 2.
I20240503 08:01:05.590420 3842 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.592033 3734 leader_election.cc:302] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, 68a5c03f0bc8469c9a6f1b31546a3ba3; no voters:
I20240503 08:01:05.592232 4510 raft_consensus.cc:2781] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:05.592346 4510 raft_consensus.cc:686] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:05.592500 4510 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:05.592808 3842 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 2.
I20240503 08:01:05.593928 3662 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 1 to 2, leader changed from 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.598973 4518 ts_tablet_manager.cc:902] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:05.599500 4518 tablet_copy_client.cc:308] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:05.599869 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet 6dbb6ccaa8d144c48404a95066a11ab1 from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:45746)
I20240503 08:01:05.599987 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet 6dbb6ccaa8d144c48404a95066a11ab1 from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:45746: session id = a08972a8b584403aaf1a7fb51b775a79-6dbb6ccaa8d144c48404a95066a11ab1
I20240503 08:01:05.600988 3855 tablet_copy_source_session.cc:205] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:05.601714 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbb6ccaa8d144c48404a95066a11ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:05.603706 4518 tablet_copy_client.cc:783] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:05.603870 4518 tablet_copy_client.cc:647] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:05.604882 4522 tablet_copy_client.cc:1002] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:05.732914 4523 raft_consensus.cc:979] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.733039 4523 raft_consensus.cc:1067] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.733433 3766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34962
I20240503 08:01:05.733572 3766 raft_consensus.cc:482] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.733639 3766 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.734616 3766 raft_consensus.cc:504] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.735327 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:05.735455 3842 raft_consensus.cc:3032] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:01:05.735513 3842 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:05.735621 3766 leader_election.cc:288] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249)
I20240503 08:01:05.735663 3842 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.735848 3842 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.736989 3842 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:05.737270 3733 leader_election.cc:302] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:05.737457 4510 raft_consensus.cc:2781] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:01:05.737546 4510 raft_consensus.cc:686] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:05.737681 4510 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.739408 3662 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 2 to 3, leader changed from e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.739660 4310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "a08972a8b584403aaf1a7fb51b775a79"
I20240503 08:01:05.739760 4310 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.740636 4310 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:05.867041 4452 raft_consensus.cc:979] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8: : Instructing follower be55fde3594e417faf23e55a23944033 to start an election
I20240503 08:01:05.867173 4523 raft_consensus.cc:1067] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Signalling peer be55fde3594e417faf23e55a23944033 to start an election
I20240503 08:01:05.867574 4159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff"
dest_uuid: "be55fde3594e417faf23e55a23944033"
from {username='slave'} at 127.0.0.1:53258
I20240503 08:01:05.867697 4159 raft_consensus.cc:482] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.867751 4159 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.868866 4159 raft_consensus.cc:504] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:05.869366 4159 leader_election.cc:288] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [CANDIDATE]: Term 2 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.869549 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "be55fde3594e417faf23e55a23944033" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:05.869690 3844 raft_consensus.cc:3032] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:05.869745 3844 raft_consensus.cc:729] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:05.869887 3844 consensus_queue.cc:260] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:05.870023 3844 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.870970 3844 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be55fde3594e417faf23e55a23944033 in term 2.
I20240503 08:01:05.871582 4123 leader_election.cc:302] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be55fde3594e417faf23e55a23944033, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:05.871905 4528 raft_consensus.cc:2781] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:05.872078 4528 raft_consensus.cc:686] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 2 LEADER]: Becoming Leader. State: Replica: be55fde3594e417faf23e55a23944033, State: Running, Role: LEADER
I20240503 08:01:05.872489 4528 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:05.873271 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9811bcfc2e9f495ca6115718712ac4ff" candidate_uuid: "be55fde3594e417faf23e55a23944033" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:05.873383 3916 raft_consensus.cc:3037] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.874132 3662 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 reported cstate change: term changed from 1 to 2, leader changed from e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) to be55fde3594e417faf23e55a23944033 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "be55fde3594e417faf23e55a23944033" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:05.874431 3916 raft_consensus.cc:2454] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be55fde3594e417faf23e55a23944033 in term 2.
I20240503 08:01:05.919586 4456 raft_consensus.cc:979] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.919710 4456 raft_consensus.cc:1067] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:05.920117 3767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34952
I20240503 08:01:05.920280 3767 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.920348 3767 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.921476 3767 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:05.921867 3767 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Requested vote from peers 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:05.922165 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:05.922233 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:05.922287 3916 raft_consensus.cc:3032] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:01:05.922329 3916 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:05.922350 3844 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.922477 3916 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:05.922669 3916 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:05.923216 3844 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:05.923399 3916 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 3.
I20240503 08:01:05.923470 3733 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:05.923645 4483 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:01:05.923731 4483 raft_consensus.cc:686] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:05.923854 4483 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:05.925594 3662 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 2 to 3, leader changed from 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:05.930770 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:05.930907 3767 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
I20240503 08:01:05.936458 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:05.936646 3767 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
I20240503 08:01:05.937640 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:05.937757 3767 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership
W20240503 08:01:05.938118 3714 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for c59c68fd5da24311a940047a49eb36ec already in progress
W20240503 08:01:05.938165 3714 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for c59c68fd5da24311a940047a49eb36ec already in progress
I20240503 08:01:05.954731 4425 raft_consensus.cc:979] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3: : Instructing follower a08972a8b584403aaf1a7fb51b775a79 to start an election
I20240503 08:01:05.954852 4425 raft_consensus.cc:1067] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Signalling peer a08972a8b584403aaf1a7fb51b775a79 to start an election
I20240503 08:01:05.955228 4310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f"
dest_uuid: "a08972a8b584403aaf1a7fb51b775a79"
from {username='slave'} at 127.0.0.1:43752
I20240503 08:01:05.955384 4310 raft_consensus.cc:482] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:05.955459 4310 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.956560 4310 raft_consensus.cc:504] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.957015 4310 leader_election.cc:288] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:05.957613 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "a08972a8b584403aaf1a7fb51b775a79" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:05.957738 3916 raft_consensus.cc:3032] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:05.957789 3916 raft_consensus.cc:729] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:05.957962 3916 consensus_queue.cc:260] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.958161 3916 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.959235 3916 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08972a8b584403aaf1a7fb51b775a79 in term 2.
I20240503 08:01:05.959564 4275 leader_election.cc:302] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, a08972a8b584403aaf1a7fb51b775a79; no voters:
I20240503 08:01:05.959899 4534 raft_consensus.cc:2781] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:05.960104 4534 raft_consensus.cc:686] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEADER]: Becoming Leader. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Running, Role: LEADER
I20240503 08:01:05.960268 4534 consensus_queue.cc:237] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:05.961753 3662 catalog_manager.cc:5484] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 reported cstate change: term changed from 1 to 2, leader changed from 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) to a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6). New cstate: current_term: 2 leader_uuid: "a08972a8b584403aaf1a7fb51b775a79" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:05.962785 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5899a1db66d148be808923d5b15aa34f" candidate_uuid: "a08972a8b584403aaf1a7fb51b775a79" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:05.962905 3767 raft_consensus.cc:3037] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:05.963730 3767 raft_consensus.cc:2454] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08972a8b584403aaf1a7fb51b775a79 in term 2.
I20240503 08:01:06.021227 3918 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:01:06.021636 4511 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:06.024533 3766 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:01:06.025014 4538 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240503 08:01:06.031301 4198 consensus_peers.cc:485] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:01:06.039048 3732 consensus_peers.cc:485] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20be03fba1ee443dba1709cbc69707bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:06.095405 3844 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:01:06.095855 4483 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:06.098702 3918 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:01:06.099201 4483 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:06.233141 3842 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:01:06.233532 4483 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:06.235695 4310 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:01:06.236063 4483 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:06.270088 3918 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer be55fde3594e417faf23e55a23944033: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:01:06.270440 4528 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:06.273447 3844 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Refusing update from remote peer be55fde3594e417faf23e55a23944033: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:01:06.273780 4529 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:06.344357 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:01:06.344825 4543 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:06.346686 4543 raft_consensus.cc:979] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:06.346784 4543 raft_consensus.cc:1067] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Signalling peer 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:06.347517 3844 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240503 08:01:06.347676 3735 consensus_peers.cc:485] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:06.347791 3916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
from {username='slave'} at 127.0.0.1:58522
I20240503 08:01:06.348055 3916 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:06.348140 3916 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:06.349323 3916 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:06.349732 3916 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Requested vote from peers e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191), 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:06.348145 4483 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.002s
I20240503 08:01:06.350013 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:06.350091 3844 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:06.350620 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:06.350709 3767 raft_consensus.cc:3032] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:01:06.350751 3767 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:06.350879 3767 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:06.351043 3844 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 4.
I20240503 08:01:06.351066 3767 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:06.351186 3842 raft_consensus.cc:1226] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Rejecting Update request from peer 44f0a2c3ee394d70ad7145e6b04ad926 for earlier term 3. Current term is 4. Ops: [3.30-3.30]
I20240503 08:01:06.351505 3884 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:06.351681 4546 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:01:06.351783 4546 raft_consensus.cc:686] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:06.351940 3767 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 4.
I20240503 08:01:06.351955 4546 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:06.353386 3662 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 3 to 4, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:06.395262 3918 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer a08972a8b584403aaf1a7fb51b775a79: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:06.395643 4535 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:06.398104 3767 raft_consensus.cc:1261] T 5899a1db66d148be808923d5b15aa34f P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer a08972a8b584403aaf1a7fb51b775a79: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:06.399164 4535 consensus_queue.cc:1035] T 5899a1db66d148be808923d5b15aa34f P a08972a8b584403aaf1a7fb51b775a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:06.506767 4559 ts_tablet_manager.cc:902] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:06.507396 4559 tablet_copy_client.cc:308] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:32789
I20240503 08:01:06.507809 3779 tablet_copy_service.cc:140] P 44f0a2c3ee394d70ad7145e6b04ad926: Received BeginTabletCopySession request for tablet 20be03fba1ee443dba1709cbc69707bd from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:35004)
I20240503 08:01:06.507912 3779 tablet_copy_service.cc:161] P 44f0a2c3ee394d70ad7145e6b04ad926: Beginning new tablet copy session on tablet 20be03fba1ee443dba1709cbc69707bd from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:35004: session id = be55fde3594e417faf23e55a23944033-20be03fba1ee443dba1709cbc69707bd
I20240503 08:01:06.508855 3779 tablet_copy_source_session.cc:205] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:06.509652 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20be03fba1ee443dba1709cbc69707bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:06.511979 4559 tablet_copy_client.cc:783] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:06.512135 4559 tablet_copy_client.cc:647] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:06.514477 4563 tablet_copy_client.cc:1002] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:06.758124 3842 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:06.758603 4456 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:06.761701 3767 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
W20240503 08:01:06.761987 3886 consensus_peers.cc:485] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:06.762465 4425 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:07.455358 4486 tablet_copy_client.cc:523] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:07.457046 4486 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Bootstrap starting.
I20240503 08:01:07.481405 4486 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:07.481767 4486 tablet_bootstrap.cc:492] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Bootstrap complete.
I20240503 08:01:07.481986 4486 ts_tablet_manager.cc:1367] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.003s
I20240503 08:01:07.482472 4486 raft_consensus.cc:348] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:07.482617 4486 raft_consensus.cc:729] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be55fde3594e417faf23e55a23944033, State: Initialized, Role: LEARNER
I20240503 08:01:07.482725 4486 consensus_queue.cc:260] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:07.483037 4486 ts_tablet_manager.cc:1398] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:01:07.483605 3779 tablet_copy_service.cc:342] P 44f0a2c3ee394d70ad7145e6b04ad926: Request end of tablet copy session be55fde3594e417faf23e55a23944033-88348c5624c94fbcb125f112e65bdb34 received from {username='slave'} at 127.0.0.1:35004
I20240503 08:01:07.483691 3779 tablet_copy_service.cc:433] P 44f0a2c3ee394d70ad7145e6b04ad926: ending tablet copy session be55fde3594e417faf23e55a23944033-88348c5624c94fbcb125f112e65bdb34 on tablet 88348c5624c94fbcb125f112e65bdb34 with peer be55fde3594e417faf23e55a23944033
I20240503 08:01:07.502264 4491 tablet_copy_client.cc:523] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:07.503768 4491 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap starting.
I20240503 08:01:07.525172 4491 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:07.525467 4491 tablet_bootstrap.cc:492] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap complete.
I20240503 08:01:07.525665 4491 ts_tablet_manager.cc:1367] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 08:01:07.526084 4491 raft_consensus.cc:348] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:07.526189 4491 raft_consensus.cc:729] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Initialized, Role: LEARNER
I20240503 08:01:07.526289 4491 consensus_queue.cc:260] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:07.526592 4491 ts_tablet_manager.cc:1398] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:01:07.527009 3930 tablet_copy_service.cc:342] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Request end of tablet copy session a08972a8b584403aaf1a7fb51b775a79-26e0121d8d964328ae012d07b34c26e7 received from {username='slave'} at 127.0.0.1:58570
I20240503 08:01:07.527097 3930 tablet_copy_service.cc:433] P 68a5c03f0bc8469c9a6f1b31546a3ba3: ending tablet copy session a08972a8b584403aaf1a7fb51b775a79-26e0121d8d964328ae012d07b34c26e7 on tablet 26e0121d8d964328ae012d07b34c26e7 with peer a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:07.529681 4310 raft_consensus.cc:1203] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 08:01:07.546205 4496 tablet_copy_client.cc:523] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:07.547425 4496 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap starting.
I20240503 08:01:07.547811 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:01:07.549921 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:01:07.550046 3767 raft_consensus.cc:595] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Received request to transfer leadership to TS e01f09ad924f45c699fb12e45a597ac8
I20240503 08:01:07.550841 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:01:07.550946 3767 raft_consensus.cc:595] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Received request to transfer leadership to TS e01f09ad924f45c699fb12e45a597ac8
I20240503 08:01:07.551908 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:51122
I20240503 08:01:07.552003 3916 raft_consensus.cc:595] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:07.552321 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:01:07.552428 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:01:07.569664 4496 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:07.569979 4496 tablet_bootstrap.cc:492] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap complete.
I20240503 08:01:07.570175 4496 ts_tablet_manager.cc:1367] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240503 08:01:07.570654 4496 raft_consensus.cc:348] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.570773 4496 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f8e707d0d0eb4812b9e22be35d86a152, State: Initialized, Role: LEARNER
I20240503 08:01:07.570845 4496 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.571132 4496 ts_tablet_manager.cc:1398] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:01:07.571549 3930 tablet_copy_service.cc:342] P 68a5c03f0bc8469c9a6f1b31546a3ba3: Request end of tablet copy session f8e707d0d0eb4812b9e22be35d86a152-c59c68fd5da24311a940047a49eb36ec received from {username='slave'} at 127.0.0.1:51120
I20240503 08:01:07.571630 3930 tablet_copy_service.cc:433] P 68a5c03f0bc8469c9a6f1b31546a3ba3: ending tablet copy session f8e707d0d0eb4812b9e22be35d86a152-c59c68fd5da24311a940047a49eb36ec on tablet c59c68fd5da24311a940047a49eb36ec with peer f8e707d0d0eb4812b9e22be35d86a152
I20240503 08:01:07.577373 4505 tablet_copy_client.cc:523] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:07.578704 4505 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap starting.
I20240503 08:01:07.599740 4505 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:07.600049 4505 tablet_bootstrap.cc:492] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Bootstrap complete.
I20240503 08:01:07.600268 4505 ts_tablet_manager.cc:1367] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.005s
I20240503 08:01:07.600782 4505 raft_consensus.cc:348] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.600936 4505 raft_consensus.cc:729] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f8e707d0d0eb4812b9e22be35d86a152, State: Initialized, Role: LEARNER
I20240503 08:01:07.601186 4505 consensus_queue.cc:260] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.601500 4505 ts_tablet_manager.cc:1398] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:01:07.601949 3855 tablet_copy_service.cc:342] P e01f09ad924f45c699fb12e45a597ac8: Request end of tablet copy session f8e707d0d0eb4812b9e22be35d86a152-a8ecce677bc4429bbab195d09e822bc7 received from {username='slave'} at 127.0.0.1:45740
I20240503 08:01:07.602032 3855 tablet_copy_service.cc:433] P e01f09ad924f45c699fb12e45a597ac8: ending tablet copy session f8e707d0d0eb4812b9e22be35d86a152-a8ecce677bc4429bbab195d09e822bc7 on tablet a8ecce677bc4429bbab195d09e822bc7 with peer f8e707d0d0eb4812b9e22be35d86a152
I20240503 08:01:07.605150 4518 tablet_copy_client.cc:523] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:07.606498 4518 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap starting.
I20240503 08:01:07.627455 4518 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:07.627760 4518 tablet_bootstrap.cc:492] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Bootstrap complete.
I20240503 08:01:07.627971 4518 ts_tablet_manager.cc:1367] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20240503 08:01:07.628466 4518 raft_consensus.cc:348] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:07.628594 4518 raft_consensus.cc:729] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Initialized, Role: LEARNER
I20240503 08:01:07.628697 4518 consensus_queue.cc:260] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:07.628994 4518 ts_tablet_manager.cc:1398] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 08:01:07.629448 3855 tablet_copy_service.cc:342] P e01f09ad924f45c699fb12e45a597ac8: Request end of tablet copy session a08972a8b584403aaf1a7fb51b775a79-6dbb6ccaa8d144c48404a95066a11ab1 received from {username='slave'} at 127.0.0.1:45746
I20240503 08:01:07.629530 3855 tablet_copy_service.cc:433] P e01f09ad924f45c699fb12e45a597ac8: ending tablet copy session a08972a8b584403aaf1a7fb51b775a79-6dbb6ccaa8d144c48404a95066a11ab1 on tablet 6dbb6ccaa8d144c48404a95066a11ab1 with peer a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:07.631590 4546 raft_consensus.cc:979] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:07.631693 4425 raft_consensus.cc:1067] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:07.632048 3767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34952
I20240503 08:01:07.632181 3767 raft_consensus.cc:482] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:07.632239 3767 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:01:07.633203 3767 raft_consensus.cc:504] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.633545 3767 leader_election.cc:288] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 5 election: Requested vote from peers 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:07.633846 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:07.633950 3916 raft_consensus.cc:3032] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:01:07.633991 3916 raft_consensus.cc:729] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:07.634061 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59c68fd5da24311a940047a49eb36ec" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:07.634167 3842 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:01:07.634124 3916 consensus_queue.cc:260] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.634374 3916 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:01:07.635272 3916 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 5.
I20240503 08:01:07.635272 3842 raft_consensus.cc:2454] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 5.
I20240503 08:01:07.635576 3733 leader_election.cc:302] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:07.635740 4575 raft_consensus.cc:2781] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:01:07.635828 4575 raft_consensus.cc:686] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:07.635954 4575 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:07.637460 3662 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 4 to 5, leader changed from 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 5 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:07.663494 4159 raft_consensus.cc:3037] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 1 LEARNER]: Advancing to term 2
I20240503 08:01:07.701786 4234 raft_consensus.cc:1203] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 08:01:07.744310 4510 raft_consensus.cc:979] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:07.744447 4510 raft_consensus.cc:1067] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Signalling peer e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:07.744832 3842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:45694
I20240503 08:01:07.744968 3842 raft_consensus.cc:482] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:07.745031 3842 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:07.746001 3842 raft_consensus.cc:504] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:07.746338 3842 leader_election.cc:288] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 3 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363)
I20240503 08:01:07.746603 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:07.746716 3767 raft_consensus.cc:3032] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:01:07.746766 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:07.746817 3767 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:07.746914 3918 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:07.747014 3767 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:07.747216 3767 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:01:07.747903 3918 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 3.
I20240503 08:01:07.748076 3767 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 3.
I20240503 08:01:07.748241 3810 leader_election.cc:302] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:07.748412 4579 raft_consensus.cc:2781] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:01:07.748515 4579 raft_consensus.cc:686] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:07.748642 4579 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:07.750165 3662 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 2 to 3, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:07.754519 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:07.754628 3842 raft_consensus.cc:595] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Received request to transfer leadership
I20240503 08:01:07.758754 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:07.758898 3842 raft_consensus.cc:595] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Received request to transfer leadership
I20240503 08:01:07.759712 3842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:07.759829 3842 raft_consensus.cc:595] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Received request to transfer leadership
W20240503 08:01:07.760190 3714 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 20be03fba1ee443dba1709cbc69707bd already in progress
W20240503 08:01:07.760262 3714 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 20be03fba1ee443dba1709cbc69707bd already in progress
I20240503 08:01:07.808650 4575 raft_consensus.cc:979] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:07.808779 4575 raft_consensus.cc:1067] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Signalling peer e01f09ad924f45c699fb12e45a597ac8 to start an election
I20240503 08:01:07.809175 3843 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
from {username='slave'} at 127.0.0.1:45694
I20240503 08:01:07.809314 3843 raft_consensus.cc:482] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:07.809389 3843 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:07.810493 3843 raft_consensus.cc:504] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:07.810820 3843 leader_election.cc:288] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 4 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249)
I20240503 08:01:07.811093 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:07.811177 3766 raft_consensus.cc:3032] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:01:07.811203 3766 raft_consensus.cc:729] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:07.811303 3766 consensus_queue.cc:260] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:07.811446 3766 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:07.812286 3766 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 4.
I20240503 08:01:07.812477 4310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06177450ba6d45f48d6c3cf937737158" candidate_uuid: "e01f09ad924f45c699fb12e45a597ac8" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "a08972a8b584403aaf1a7fb51b775a79"
I20240503 08:01:07.812520 3809 leader_election.cc:302] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:07.812583 4310 raft_consensus.cc:3037] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:07.812701 4579 raft_consensus.cc:2781] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:01:07.812772 4579 raft_consensus.cc:686] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [term 4 LEADER]: Becoming Leader. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:07.812904 4579 consensus_queue.cc:237] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:07.813302 4310 raft_consensus.cc:2454] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f09ad924f45c699fb12e45a597ac8 in term 4.
I20240503 08:01:07.814539 3662 catalog_manager.cc:5484] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: term changed from 3 to 4, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:07.950523 4310 raft_consensus.cc:1203] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 08:01:08.012384 4425 raft_consensus.cc:1050] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: attempting to promote NON_VOTER a08972a8b584403aaf1a7fb51b775a79 to VOTER
I20240503 08:01:08.012758 4425 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:08.013820 4310 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:08.013947 3843 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:08.014325 4546 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.014474 4425 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.014679 3766 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:08.015002 4425 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.016902 4425 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.017431 3842 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.018399 3766 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.018868 4310 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.019451 3662 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 28 to 29, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.022729 3918 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:08.023932 4310 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:08.024133 3766 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:08.024680 4589 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.025139 4589 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.026104 4546 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.026558 4310 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.028019 3652 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26e0121d8d964328ae012d07b34c26e7 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:08.028443 3661 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.028451 3766 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.030663 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e0121d8d964328ae012d07b34c26e7 with delete_type TABLET_DATA_TOMBSTONED (TS e01f09ad924f45c699fb12e45a597ac8 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:08.031023 4594 tablet_replica.cc:330] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: stopping tablet replica
I20240503 08:01:08.031333 4594 raft_consensus.cc:2229] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:08.031589 4594 raft_consensus.cc:2258] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:08.032197 4594 ts_tablet_manager.cc:1875] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:08.034183 4594 ts_tablet_manager.cc:1888] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240503 08:01:08.034261 4594 log.cc:1196] T 26e0121d8d964328ae012d07b34c26e7 P e01f09ad924f45c699fb12e45a597ac8: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/wals/26e0121d8d964328ae012d07b34c26e7
I20240503 08:01:08.034787 3651 catalog_manager.cc:4830] TS e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191): tablet 26e0121d8d964328ae012d07b34c26e7 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:08.071231 4595 raft_consensus.cc:1050] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: attempting to promote NON_VOTER be55fde3594e417faf23e55a23944033 to VOTER
I20240503 08:01:08.071784 4595 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:08.073421 3766 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:01:08.073423 3918 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:01:08.073439 4159 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 LEARNER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:01:08.073899 4598 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.074117 4597 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.074335 4595 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.076421 4598 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.076922 3918 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.078626 3657 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 31 to 33, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f8e707d0d0eb4812b9e22be35d86a152" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.080003 3766 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.080483 4159 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.082541 3842 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:01:08.082912 4575 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:08.085088 4234 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:08.085971 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:01:08.086094 4234 raft_consensus.cc:3037] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Advancing to term 5
I20240503 08:01:08.086572 3766 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 08:01:08.087134 4510 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:08.087184 4234 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 LEARNER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:01:08.087740 4575 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.088290 4159 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 08:01:08.088867 4564 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:01:08.089125 4597 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:01:08.090801 4575 raft_consensus.cc:1050] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
I20240503 08:01:08.091194 4575 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:08.091332 4610 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.091630 4159 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.092480 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 08:01:08.092526 3843 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 08:01:08.092938 4510 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.093089 4234 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 LEARNER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 08:01:08.093374 4510 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.093639 4510 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.094215 3766 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.094636 3662 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 reported cstate change: config changed from index 33 to 34, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "f8e707d0d0eb4812b9e22be35d86a152" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.096031 3713 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88348c5624c94fbcb125f112e65bdb34 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:08.096189 4575 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 29 to 33, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.097471 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet 88348c5624c94fbcb125f112e65bdb34 with delete_type TABLET_DATA_TOMBSTONED (TS 68a5c03f0bc8469c9a6f1b31546a3ba3 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:58516
I20240503 08:01:08.098083 3918 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.098456 4234 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.098649 4612 tablet_replica.cc:330] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: stopping tablet replica
I20240503 08:01:08.098824 4612 raft_consensus.cc:2229] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:08.099098 4612 raft_consensus.cc:2258] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:08.099145 3843 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 29 to 33, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.100353 3661 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index 29 to 33, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.101353 4612 ts_tablet_manager.cc:1875] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:08.104655 4612 ts_tablet_manager.cc:1888] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240503 08:01:08.104732 4612 log.cc:1196] T 88348c5624c94fbcb125f112e65bdb34 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/wals/88348c5624c94fbcb125f112e65bdb34
I20240503 08:01:08.105196 3652 catalog_manager.cc:4830] TS 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363): tablet 88348c5624c94fbcb125f112e65bdb34 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:08.190379 4566 raft_consensus.cc:1050] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8: attempting to promote NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 to VOTER
I20240503 08:01:08.190881 4566 consensus_queue.cc:237] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
W20240503 08:01:08.192121 3808 consensus_peers.cc:485] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:08.192162 4234 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEARNER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:08.192373 3766 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:08.192317 3918 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:01:08.192863 4615 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.193058 4579 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.193197 4566 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.195044 4566 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.195726 3766 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.197367 3918 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.197981 3662 catalog_manager.cc:5484] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.198577 4234 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.200899 3918 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:01:08.201591 4579 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.201846 3843 consensus_queue.cc:237] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:08.203054 3916 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:08.203390 4234 raft_consensus.cc:1261] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:01:08.203493 4617 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.203573 4579 raft_consensus.cc:979] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: : Instructing follower 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:08.203639 3766 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:01:08.203644 4579 raft_consensus.cc:1067] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Signalling peer 68a5c03f0bc8469c9a6f1b31546a3ba3 to start an election
I20240503 08:01:08.203819 4615 consensus_queue.cc:1035] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.204296 3918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
from {username='slave'} at 127.0.0.1:58524
I20240503 08:01:08.204408 3918 raft_consensus.cc:482] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:08.204473 3918 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:08.205376 4579 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.205399 3918 raft_consensus.cc:504] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:08.205739 3918 leader_election.cc:288] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:08.206002 4579 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P e01f09ad924f45c699fb12e45a597ac8 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.206382 3918 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.207681 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:08.207742 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20be03fba1ee443dba1709cbc69707bd" candidate_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:08.207837 3842 raft_consensus.cc:3032] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:01:08.207883 3842 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:08.208034 3842 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:08.208237 3842 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:08.209160 3842 raft_consensus.cc:2454] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a5c03f0bc8469c9a6f1b31546a3ba3 in term 4.
I20240503 08:01:08.209452 3884 leader_election.cc:302] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68a5c03f0bc8469c9a6f1b31546a3ba3, e01f09ad924f45c699fb12e45a597ac8; no voters: 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:08.209623 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8ecce677bc4429bbab195d09e822bc7 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:08.209986 3662 catalog_manager.cc:5484] T a8ecce677bc4429bbab195d09e822bc7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New cstate: current_term: 2 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.209640 4546 raft_consensus.cc:2781] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:01:08.210597 4546 raft_consensus.cc:686] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Becoming Leader. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:08.210727 4546 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:08.213089 3755 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ecce677bc4429bbab195d09e822bc7 with delete_type TABLET_DATA_TOMBSTONED (TS 44f0a2c3ee394d70ad7145e6b04ad926 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:08.213449 4623 tablet_replica.cc:330] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: stopping tablet replica
I20240503 08:01:08.213579 4623 raft_consensus.cc:2229] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:08.213719 4623 raft_consensus.cc:2258] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:08.214273 4623 ts_tablet_manager.cc:1875] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:08.214574 3662 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: term changed from 3 to 4, leader changed from e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) to 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:08.216060 4234 raft_consensus.cc:2932] T a8ecce677bc4429bbab195d09e822bc7 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:08.217098 4623 ts_tablet_manager.cc:1888] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 08:01:08.217164 4623 log.cc:1196] T a8ecce677bc4429bbab195d09e822bc7 P 44f0a2c3ee394d70ad7145e6b04ad926: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-0-root/wals/a8ecce677bc4429bbab195d09e822bc7
I20240503 08:01:08.218618 3651 catalog_manager.cc:4830] TS 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789): tablet a8ecce677bc4429bbab195d09e822bc7 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:08.398348 4310 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P a08972a8b584403aaf1a7fb51b775a79 [term 4 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 08:01:08.398802 4566 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.401111 3767 raft_consensus.cc:1261] T 06177450ba6d45f48d6c3cf937737158 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 08:01:08.401471 4617 consensus_queue.cc:1035] T 06177450ba6d45f48d6c3cf937737158 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:01:08.415788 4579 raft_consensus.cc:1050] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: attempting to promote NON_VOTER a08972a8b584403aaf1a7fb51b775a79 to VOTER
I20240503 08:01:08.416185 4579 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:08.417291 3767 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:08.417335 4310 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 LEARNER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:08.417362 3918 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:08.417688 4615 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:08.417842 4617 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:08.418012 4579 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:08.419924 4579 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.420321 4310 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.421761 3767 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.422349 3918 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.422509 3661 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 27 to 28, a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.425678 3843 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } }
I20240503 08:01:08.426818 4310 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:08.426818 3767 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 08:01:08.427253 4579 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.427603 4579 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.429001 4566 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.429536 4310 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.430473 3767 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } }
I20240503 08:01:08.430673 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6dbb6ccaa8d144c48404a95066a11ab1 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:08.431169 3657 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 28 to 29, VOTER 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:08.433822 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbb6ccaa8d144c48404a95066a11ab1 with delete_type TABLET_DATA_TOMBSTONED (TS 68a5c03f0bc8469c9a6f1b31546a3ba3 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:58516
I20240503 08:01:08.434023 4612 tablet_replica.cc:330] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: stopping tablet replica
I20240503 08:01:08.434149 4612 raft_consensus.cc:2229] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:08.434293 4612 raft_consensus.cc:2258] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:08.434983 4612 ts_tablet_manager.cc:1875] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:08.436698 4612 ts_tablet_manager.cc:1888] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 08:01:08.436761 4612 log.cc:1196] T 6dbb6ccaa8d144c48404a95066a11ab1 P 68a5c03f0bc8469c9a6f1b31546a3ba3: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-2-root/wals/6dbb6ccaa8d144c48404a95066a11ab1
I20240503 08:01:08.437255 3652 catalog_manager.cc:4830] TS 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363): tablet 6dbb6ccaa8d144c48404a95066a11ab1 (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:08.515034 4559 tablet_copy_client.cc:523] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:01:08.516494 4559 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Bootstrap starting.
I20240503 08:01:08.535450 4559 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:01:08.535750 4559 tablet_bootstrap.cc:492] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Bootstrap complete.
I20240503 08:01:08.535933 4559 ts_tablet_manager.cc:1367] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 08:01:08.536398 4559 raft_consensus.cc:348] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:08.536511 4559 raft_consensus.cc:729] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: be55fde3594e417faf23e55a23944033, State: Initialized, Role: LEARNER
I20240503 08:01:08.536590 4559 consensus_queue.cc:260] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:08.536865 4559 ts_tablet_manager.cc:1398] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:01:08.537480 3779 tablet_copy_service.cc:342] P 44f0a2c3ee394d70ad7145e6b04ad926: Request end of tablet copy session be55fde3594e417faf23e55a23944033-20be03fba1ee443dba1709cbc69707bd received from {username='slave'} at 127.0.0.1:35004
I20240503 08:01:08.537562 3779 tablet_copy_service.cc:433] P 44f0a2c3ee394d70ad7145e6b04ad926: ending tablet copy session be55fde3594e417faf23e55a23944033-20be03fba1ee443dba1709cbc69707bd on tablet 20be03fba1ee443dba1709cbc69707bd with peer be55fde3594e417faf23e55a23944033
I20240503 08:01:08.638501 3842 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:01:08.638909 4589 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.641475 4159 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 2 LEARNER]: Advancing to term 4
I20240503 08:01:08.641680 3767 raft_consensus.cc:3037] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:01:08.642769 4159 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 4 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:01:08.642786 3767 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:01:08.643244 4546 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:01:08.643365 4589 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:08.645404 4546 raft_consensus.cc:1050] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3: attempting to promote NON_VOTER be55fde3594e417faf23e55a23944033 to VOTER
I20240503 08:01:08.645805 4546 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 29, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:08.647074 3767 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:08.647073 3842 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:08.647460 4636 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:08.647526 4159 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 4 LEARNER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:01:08.647651 4589 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:08.647902 4589 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:08.649312 4546 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 27 to 31, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.649740 3767 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 27 to 31, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.649737 4159 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 27 to 31, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.651551 3842 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 27 to 31, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.651988 3662 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 reported cstate change: config changed from index 27 to 31, be55fde3594e417faf23e55a23944033 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.655516 3918 consensus_queue.cc:237] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:08.656548 3767 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:01:08.656565 4159 raft_consensus.cc:1261] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 4 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 08:01:08.656893 4636 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:08.657059 4589 consensus_queue.cc:1035] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:08.658430 4633 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.658926 3767 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.658922 4159 raft_consensus.cc:2932] T 20be03fba1ee443dba1709cbc69707bd P be55fde3594e417faf23e55a23944033 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.660336 3652 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20be03fba1ee443dba1709cbc69707bd with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:08.661299 3661 catalog_manager.cc:5484] T 20be03fba1ee443dba1709cbc69707bd P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index 31 to 32, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New cstate: current_term: 4 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } }
I20240503 08:01:08.663828 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 20be03fba1ee443dba1709cbc69707bd with delete_type TABLET_DATA_TOMBSTONED (TS e01f09ad924f45c699fb12e45a597ac8 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:08.664268 4637 tablet_replica.cc:330] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: stopping tablet replica
I20240503 08:01:08.664412 4637 raft_consensus.cc:2229] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:08.664602 4637 raft_consensus.cc:2258] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:08.665139 4637 ts_tablet_manager.cc:1875] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:08.667014 4637 ts_tablet_manager.cc:1888] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.31
I20240503 08:01:08.667091 4637 log.cc:1196] T 20be03fba1ee443dba1709cbc69707bd P e01f09ad924f45c699fb12e45a597ac8: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/wals/20be03fba1ee443dba1709cbc69707bd
I20240503 08:01:08.667518 3651 catalog_manager.cc:4830] TS e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191): tablet 20be03fba1ee443dba1709cbc69707bd (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:09.435287 3918 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:09.436688 3767 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240503 08:01:09.436740 3886 consensus_peers.cc:485] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26e0121d8d964328ae012d07b34c26e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.436688 4310 raft_consensus.cc:1261] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Refusing update from remote peer 68a5c03f0bc8469c9a6f1b31546a3ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 08:01:09.437160 4640 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:09.437359 4633 consensus_queue.cc:1035] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:01:09.438989 4633 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.439342 4310 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.439713 3767 raft_consensus.cc:2932] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.441498 3659 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 30 to 31, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:09.450858 4159 consensus_queue.cc:237] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:09.452306 3918 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Refusing update from remote peer be55fde3594e417faf23e55a23944033: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:01:09.452481 3842 raft_consensus.cc:1261] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Refusing update from remote peer be55fde3594e417faf23e55a23944033: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:01:09.452840 4650 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:09.453037 4651 consensus_queue.cc:1035] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:01:09.454680 4650 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.455061 3842 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.455253 3918 raft_consensus.cc:2932] T 9811bcfc2e9f495ca6115718712ac4ff P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.457211 3661 catalog_manager.cc:5484] T 9811bcfc2e9f495ca6115718712ac4ff P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 30 to 32, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 2 leader_uuid: "be55fde3594e417faf23e55a23944033" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } attrs { replace: true } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
W20240503 08:01:09.457890 4125 consensus_peers.cc:485] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9811bcfc2e9f495ca6115718712ac4ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.458551 3767 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } }
I20240503 08:01:09.459831 4234 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240503 08:01:09.459926 3842 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240503 08:01:09.459944 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20240503 08:01:09.460376 4614 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:01:09.460561 4657 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:01:09.460861 4638 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:01:09.462836 4657 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:09.463317 3918 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:09.463264 3842 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:09.464949 4234 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:09.466465 3659 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 33 to 34. New cstate: current_term: 5 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } }
I20240503 08:01:09.466650 4234 consensus_queue.cc:237] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } }
I20240503 08:01:09.468205 4159 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 08:01:09.468528 3767 raft_consensus.cc:1261] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Refusing update from remote peer f8e707d0d0eb4812b9e22be35d86a152: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 08:01:09.469033 4595 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:01:09.469190 4662 consensus_queue.cc:1035] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:01:09.470850 3766 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:09.471803 4595 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.472965 3842 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
W20240503 08:01:09.473045 3732 consensus_peers.cc:485] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c59c68fd5da24311a940047a49eb36ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.473522 3918 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
W20240503 08:01:09.473985 4201 consensus_peers.cc:485] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 -> Peer a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249): Couldn't send request to peer a08972a8b584403aaf1a7fb51b775a79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88348c5624c94fbcb125f112e65bdb34. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.473641 4234 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240503 08:01:09.473371 4667 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
I20240503 08:01:09.474759 4614 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:01:09.475028 3842 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:09.475930 4638 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:01:09.475896 4159 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.477025 3661 catalog_manager.cc:5484] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 34 to 35, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New cstate: current_term: 2 leader_uuid: "f8e707d0d0eb4812b9e22be35d86a152" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:09.478156 3766 raft_consensus.cc:2932] T 88348c5624c94fbcb125f112e65bdb34 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: true } } }
I20240503 08:01:09.478360 4310 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:09.479552 4579 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 08:01:09.480949 3808 consensus_peers.cc:485] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dbb6ccaa8d144c48404a95066a11ab1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.480967 4638 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 LEADER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.481330 4579 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.481634 4310 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.482756 3651 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c59c68fd5da24311a940047a49eb36ec with cas_config_opid_index 34: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:01:09.483664 3918 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.484815 3767 consensus_queue.cc:237] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:09.484920 3662 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 reported cstate change: config changed from index 29 to 30, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "e01f09ad924f45c699fb12e45a597ac8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:09.480967 3766 raft_consensus.cc:1261] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Refusing update from remote peer e01f09ad924f45c699fb12e45a597ac8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 08:01:09.486344 3918 raft_consensus.cc:1261] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:09.486531 3842 raft_consensus.cc:1261] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:01:09.486776 3843 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.487334 4614 consensus_queue.cc:1035] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:09.487810 4614 consensus_queue.cc:1035] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:01:09.488525 4669 consensus_queue.cc:1035] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:01:09.488690 3662 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 reported cstate change: config changed from index 34 to 35, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New cstate: current_term: 5 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
W20240503 08:01:09.489259 3735 consensus_peers.cc:485] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779): Couldn't send request to peer f8e707d0d0eb4812b9e22be35d86a152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade6e49ee7304d79b81c03cd034a815c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.490689 3767 raft_consensus.cc:2932] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.492388 4234 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 34 to 35, NON_VOTER be55fde3594e417faf23e55a23944033 (127.0.99.4) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.493422 3767 consensus_queue.cc:237] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:09.494956 4638 raft_consensus.cc:2932] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.496699 3918 raft_consensus.cc:2932] T ade6e49ee7304d79b81c03cd034a815c P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.496706 3843 raft_consensus.cc:2932] T ade6e49ee7304d79b81c03cd034a815c P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } } }
I20240503 08:01:09.497932 4234 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240503 08:01:09.498080 3916 raft_consensus.cc:1261] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Refusing update from remote peer 44f0a2c3ee394d70ad7145e6b04ad926: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240503 08:01:09.498245 4614 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
W20240503 08:01:09.498847 3732 consensus_peers.cc:485] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 -> Peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743): Couldn't send request to peer be55fde3594e417faf23e55a23944033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c59c68fd5da24311a940047a49eb36ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:01:09.499199 4614 consensus_queue.cc:1035] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 08:01:09.499547 3661 catalog_manager.cc:5484] T ade6e49ee7304d79b81c03cd034a815c P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: config changed from index -1 to 28, NON_VOTER f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:09.499904 4614 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926 [term 5 LEADER]: Committing config change with OpId 5.36: config changed from index 35 to 36, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.500219 4234 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.502143 3651 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c59c68fd5da24311a940047a49eb36ec with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:01:09.503237 3659 catalog_manager.cc:5484] T c59c68fd5da24311a940047a49eb36ec P f8e707d0d0eb4812b9e22be35d86a152 reported cstate change: config changed from index 35 to 36, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New cstate: current_term: 5 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.503629 3918 raft_consensus.cc:2932] T c59c68fd5da24311a940047a49eb36ec P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36, VOTER e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } } }
I20240503 08:01:09.505851 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet c59c68fd5da24311a940047a49eb36ec with delete_type TABLET_DATA_TOMBSTONED (TS e01f09ad924f45c699fb12e45a597ac8 not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:45680
I20240503 08:01:09.506374 4677 tablet_replica.cc:330] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: stopping tablet replica
I20240503 08:01:09.506525 4677 raft_consensus.cc:2229] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 08:01:09.506983 4677 raft_consensus.cc:2258] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 08:01:09.507642 4677 ts_tablet_manager.cc:1875] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:01:09.509851 4677 ts_tablet_manager.cc:1888] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.35
I20240503 08:01:09.509918 4677 log.cc:1196] T c59c68fd5da24311a940047a49eb36ec P e01f09ad924f45c699fb12e45a597ac8: Deleting WAL directory at /tmp/dist-test-taskJ96kTD/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723166506157-396-0/minicluster-data/ts-1-root/wals/c59c68fd5da24311a940047a49eb36ec
I20240503 08:01:09.510380 3651 catalog_manager.cc:4830] TS e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191): tablet c59c68fd5da24311a940047a49eb36ec (table test-workload [id=65a2d6a6517f459a8ba3d2da42cb388f]) successfully deleted
I20240503 08:01:09.552677 3720 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:01:09.554528 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7"
dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
mode: GRACEFUL
new_leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:51122
I20240503 08:01:09.554678 3916 raft_consensus.cc:595] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Received request to transfer leadership to TS 44f0a2c3ee394d70ad7145e6b04ad926
I20240503 08:01:09.555371 3843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1"
dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
mode: GRACEFUL
new_leader_uuid: "a08972a8b584403aaf1a7fb51b775a79"
from {username='slave'} at 127.0.0.1:45758
I20240503 08:01:09.555478 3843 raft_consensus.cc:595] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Received request to transfer leadership to TS a08972a8b584403aaf1a7fb51b775a79
I20240503 08:01:09.556115 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
new_leader_uuid: "be55fde3594e417faf23e55a23944033"
from {username='slave'} at 127.0.0.1:34978
I20240503 08:01:09.556241 3767 raft_consensus.cc:595] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Received request to transfer leadership to TS be55fde3594e417faf23e55a23944033
I20240503 08:01:09.556504 3720 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:01:09.556620 3720 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:01:09.662389 4657 raft_consensus.cc:979] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926: : Instructing follower be55fde3594e417faf23e55a23944033 to start an election
I20240503 08:01:09.662540 4657 raft_consensus.cc:1067] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Signalling peer be55fde3594e417faf23e55a23944033 to start an election
I20240503 08:01:09.663084 4158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373"
dest_uuid: "be55fde3594e417faf23e55a23944033"
from {username='slave'} at 127.0.0.1:53244
I20240503 08:01:09.663198 4158 raft_consensus.cc:482] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:09.663259 4158 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.664877 4158 raft_consensus.cc:504] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:09.665297 4158 leader_election.cc:288] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:09.665769 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "be55fde3594e417faf23e55a23944033" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:09.665802 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6b5bb3a0ab4bbc9314b9cd3634d373" candidate_uuid: "be55fde3594e417faf23e55a23944033" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:09.665956 3766 raft_consensus.cc:3032] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:09.665977 3843 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.666008 3766 raft_consensus.cc:729] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:09.666154 3766 consensus_queue.cc:260] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:09.666296 3766 raft_consensus.cc:3037] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.667162 3843 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be55fde3594e417faf23e55a23944033 in term 2.
I20240503 08:01:09.667501 3766 raft_consensus.cc:2454] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be55fde3594e417faf23e55a23944033 in term 2.
I20240503 08:01:09.667506 4123 leader_election.cc:302] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be55fde3594e417faf23e55a23944033, e01f09ad924f45c699fb12e45a597ac8; no voters:
I20240503 08:01:09.667771 4651 raft_consensus.cc:2781] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:09.667870 4651 raft_consensus.cc:686] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [term 2 LEADER]: Becoming Leader. State: Replica: be55fde3594e417faf23e55a23944033, State: Running, Role: LEADER
I20240503 08:01:09.668045 4651 consensus_queue.cc:237] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } }
I20240503 08:01:09.669965 3661 catalog_manager.cc:5484] T cf6b5bb3a0ab4bbc9314b9cd3634d373 P be55fde3594e417faf23e55a23944033 reported cstate change: term changed from 1 to 2, leader changed from 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1) to be55fde3594e417faf23e55a23944033 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "be55fde3594e417faf23e55a23944033" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:01:09.861675 4679 ts_tablet_manager.cc:902] T 6dbb6ccaa8d144c48404a95066a11ab1 P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:09.862296 4679 tablet_copy_client.cc:308] T 6dbb6ccaa8d144c48404a95066a11ab1 P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:43191
I20240503 08:01:09.862681 3855 tablet_copy_service.cc:140] P e01f09ad924f45c699fb12e45a597ac8: Received BeginTabletCopySession request for tablet 6dbb6ccaa8d144c48404a95066a11ab1 from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:45752)
I20240503 08:01:09.862778 3855 tablet_copy_service.cc:161] P e01f09ad924f45c699fb12e45a597ac8: Beginning new tablet copy session on tablet 6dbb6ccaa8d144c48404a95066a11ab1 from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:45752: session id = be55fde3594e417faf23e55a23944033-6dbb6ccaa8d144c48404a95066a11ab1
I20240503 08:01:09.863783 3855 tablet_copy_source_session.cc:205] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:09.864651 4679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbb6ccaa8d144c48404a95066a11ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:09.866911 4679 tablet_copy_client.cc:783] T 6dbb6ccaa8d144c48404a95066a11ab1 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:09.867092 4679 tablet_copy_client.cc:647] T 6dbb6ccaa8d144c48404a95066a11ab1 P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:09.868039 4683 tablet_copy_client.cc:1002] T 6dbb6ccaa8d144c48404a95066a11ab1 P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:09.934345 4633 raft_consensus.cc:979] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3: : Instructing follower 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:09.934471 4633 raft_consensus.cc:1067] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Signalling peer 44f0a2c3ee394d70ad7145e6b04ad926 to start an election
I20240503 08:01:09.934849 3767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
from {username='slave'} at 127.0.0.1:34952
I20240503 08:01:09.934996 3767 raft_consensus.cc:482] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:09.935065 3767 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.936093 3767 raft_consensus.cc:504] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:09.936501 3767 leader_election.cc:288] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 2 election: Requested vote from peers 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3:46363), a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6:43249)
I20240503 08:01:09.936671 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3"
I20240503 08:01:09.936817 3916 raft_consensus.cc:3032] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:09.936870 3916 raft_consensus.cc:729] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68a5c03f0bc8469c9a6f1b31546a3ba3, State: Running, Role: LEADER
I20240503 08:01:09.936933 4310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7" candidate_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "a08972a8b584403aaf1a7fb51b775a79"
I20240503 08:01:09.937036 4310 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.937018 3916 consensus_queue.cc:260] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:09.937186 3916 raft_consensus.cc:3037] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:09.937765 4310 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 2.
I20240503 08:01:09.937903 3916 raft_consensus.cc:2454] T 26e0121d8d964328ae012d07b34c26e7 P 68a5c03f0bc8469c9a6f1b31546a3ba3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f0a2c3ee394d70ad7145e6b04ad926 in term 2.
I20240503 08:01:09.938095 3735 leader_election.cc:302] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, a08972a8b584403aaf1a7fb51b775a79; no voters:
I20240503 08:01:09.938366 4657 raft_consensus.cc:2781] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:09.938469 4657 raft_consensus.cc:686] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Becoming Leader. State: Replica: 44f0a2c3ee394d70ad7145e6b04ad926, State: Running, Role: LEADER
I20240503 08:01:09.938607 4657 consensus_queue.cc:237] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } }
I20240503 08:01:09.940317 3661 catalog_manager.cc:5484] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 reported cstate change: term changed from 1 to 2, leader changed from 68a5c03f0bc8469c9a6f1b31546a3ba3 (127.0.99.3) to 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a5c03f0bc8469c9a6f1b31546a3ba3" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e707d0d0eb4812b9e22be35d86a152" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:09.944944 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:09.945061 3767 raft_consensus.cc:595] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Received request to transfer leadership
I20240503 08:01:09.949587 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:09.949697 3767 raft_consensus.cc:595] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Received request to transfer leadership
W20240503 08:01:09.949903 3651 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 26e0121d8d964328ae012d07b34c26e7 already in progress [suppressed 2 similar messages]
I20240503 08:01:09.950709 3767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26e0121d8d964328ae012d07b34c26e7"
dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34948
I20240503 08:01:09.950824 3767 raft_consensus.cc:595] T 26e0121d8d964328ae012d07b34c26e7 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 LEADER]: Received request to transfer leadership
W20240503 08:01:09.951133 3714 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 26e0121d8d964328ae012d07b34c26e7 already in progress
W20240503 08:01:09.951198 3714 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 26e0121d8d964328ae012d07b34c26e7 already in progress
I20240503 08:01:09.980129 4686 ts_tablet_manager.cc:902] T c59c68fd5da24311a940047a49eb36ec P be55fde3594e417faf23e55a23944033: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
I20240503 08:01:09.980136 4685 ts_tablet_manager.cc:902] T 9811bcfc2e9f495ca6115718712ac4ff P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer be55fde3594e417faf23e55a23944033 (127.0.99.4:34743)
I20240503 08:01:09.980829 4686 tablet_copy_client.cc:308] T c59c68fd5da24311a940047a49eb36ec P be55fde3594e417faf23e55a23944033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:32789
I20240503 08:01:09.980835 4685 tablet_copy_client.cc:308] T 9811bcfc2e9f495ca6115718712ac4ff P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.4:34743
I20240503 08:01:09.981225 3779 tablet_copy_service.cc:140] P 44f0a2c3ee394d70ad7145e6b04ad926: Received BeginTabletCopySession request for tablet c59c68fd5da24311a940047a49eb36ec from peer be55fde3594e417faf23e55a23944033 ({username='slave'} at 127.0.0.1:35004)
I20240503 08:01:09.981317 3779 tablet_copy_service.cc:161] P 44f0a2c3ee394d70ad7145e6b04ad926: Beginning new tablet copy session on tablet c59c68fd5da24311a940047a49eb36ec from peer be55fde3594e417faf23e55a23944033 at {username='slave'} at 127.0.0.1:35004: session id = be55fde3594e417faf23e55a23944033-c59c68fd5da24311a940047a49eb36ec
I20240503 08:01:09.982056 3779 tablet_copy_source_session.cc:205] T c59c68fd5da24311a940047a49eb36ec P 44f0a2c3ee394d70ad7145e6b04ad926: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:09.982800 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59c68fd5da24311a940047a49eb36ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:09.985023 4169 tablet_copy_service.cc:140] P be55fde3594e417faf23e55a23944033: Received BeginTabletCopySession request for tablet 9811bcfc2e9f495ca6115718712ac4ff from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:34162)
I20240503 08:01:09.985118 4169 tablet_copy_service.cc:161] P be55fde3594e417faf23e55a23944033: Beginning new tablet copy session on tablet 9811bcfc2e9f495ca6115718712ac4ff from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:34162: session id = a08972a8b584403aaf1a7fb51b775a79-9811bcfc2e9f495ca6115718712ac4ff
I20240503 08:01:09.985145 4686 tablet_copy_client.cc:783] T c59c68fd5da24311a940047a49eb36ec P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:09.985296 4686 tablet_copy_client.cc:647] T c59c68fd5da24311a940047a49eb36ec P be55fde3594e417faf23e55a23944033: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:09.986023 4169 tablet_copy_source_session.cc:205] T 9811bcfc2e9f495ca6115718712ac4ff P be55fde3594e417faf23e55a23944033: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:09.986263 4694 tablet_copy_client.cc:1002] T c59c68fd5da24311a940047a49eb36ec P be55fde3594e417faf23e55a23944033: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:09.986780 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9811bcfc2e9f495ca6115718712ac4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:09.988514 4685 tablet_copy_client.cc:783] T 9811bcfc2e9f495ca6115718712ac4ff P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:09.988643 4685 tablet_copy_client.cc:647] T 9811bcfc2e9f495ca6115718712ac4ff P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:09.989480 4693 tablet_copy_client.cc:1002] T 9811bcfc2e9f495ca6115718712ac4ff P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:09.995100 4697 ts_tablet_manager.cc:902] T 88348c5624c94fbcb125f112e65bdb34 P a08972a8b584403aaf1a7fb51b775a79: Initiating tablet copy from peer f8e707d0d0eb4812b9e22be35d86a152 (127.0.99.5:37779)
I20240503 08:01:09.995622 4697 tablet_copy_client.cc:308] T 88348c5624c94fbcb125f112e65bdb34 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.5:37779
I20240503 08:01:09.999636 4245 tablet_copy_service.cc:140] P f8e707d0d0eb4812b9e22be35d86a152: Received BeginTabletCopySession request for tablet 88348c5624c94fbcb125f112e65bdb34 from peer a08972a8b584403aaf1a7fb51b775a79 ({username='slave'} at 127.0.0.1:37532)
I20240503 08:01:09.999749 4245 tablet_copy_service.cc:161] P f8e707d0d0eb4812b9e22be35d86a152: Beginning new tablet copy session on tablet 88348c5624c94fbcb125f112e65bdb34 from peer a08972a8b584403aaf1a7fb51b775a79 at {username='slave'} at 127.0.0.1:37532: session id = a08972a8b584403aaf1a7fb51b775a79-88348c5624c94fbcb125f112e65bdb34
I20240503 08:01:10.000583 4245 tablet_copy_source_session.cc:205] T 88348c5624c94fbcb125f112e65bdb34 P f8e707d0d0eb4812b9e22be35d86a152: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:01:10.001343 4697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88348c5624c94fbcb125f112e65bdb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:01:10.003702 4697 tablet_copy_client.cc:783] T 88348c5624c94fbcb125f112e65bdb34 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 0 data blocks...
I20240503 08:01:10.003871 4697 tablet_copy_client.cc:647] T 88348c5624c94fbcb125f112e65bdb34 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Starting download of 1 WAL segments...
I20240503 08:01:10.004842 4701 tablet_copy_client.cc:1002] T 88348c5624c94fbcb125f112e65bdb34 P a08972a8b584403aaf1a7fb51b775a79: tablet copy: Injecting latency into file download: 2000
I20240503 08:01:10.007647 4672 raft_consensus.cc:979] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8: : Instructing follower a08972a8b584403aaf1a7fb51b775a79 to start an election
I20240503 08:01:10.007742 4672 raft_consensus.cc:1067] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Signalling peer a08972a8b584403aaf1a7fb51b775a79 to start an election
I20240503 08:01:10.008069 4310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1"
dest_uuid: "a08972a8b584403aaf1a7fb51b775a79"
from {username='slave'} at 127.0.0.1:43766
I20240503 08:01:10.008219 4310 raft_consensus.cc:482] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:01:10.008289 4310 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:10.009068 4310 raft_consensus.cc:504] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:10.009541 4310 leader_election.cc:288] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [CANDIDATE]: Term 2 election: Requested vote from peers 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789), e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2:43191)
I20240503 08:01:10.009752 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "a08972a8b584403aaf1a7fb51b775a79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "44f0a2c3ee394d70ad7145e6b04ad926"
I20240503 08:01:10.009868 3766 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:10.010849 3766 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P 44f0a2c3ee394d70ad7145e6b04ad926 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08972a8b584403aaf1a7fb51b775a79 in term 2.
I20240503 08:01:10.011066 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbb6ccaa8d144c48404a95066a11ab1" candidate_uuid: "a08972a8b584403aaf1a7fb51b775a79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "e01f09ad924f45c699fb12e45a597ac8"
I20240503 08:01:10.011129 4274 leader_election.cc:302] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f0a2c3ee394d70ad7145e6b04ad926, a08972a8b584403aaf1a7fb51b775a79; no voters:
I20240503 08:01:10.011173 3843 raft_consensus.cc:3032] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:01:10.011258 3843 raft_consensus.cc:729] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e01f09ad924f45c699fb12e45a597ac8, State: Running, Role: LEADER
I20240503 08:01:10.011332 4572 raft_consensus.cc:2781] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:01:10.011418 3843 consensus_queue.cc:260] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:10.011529 4572 raft_consensus.cc:686] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [term 2 LEADER]: Becoming Leader. State: Replica: a08972a8b584403aaf1a7fb51b775a79, State: Running, Role: LEADER
I20240503 08:01:10.011646 3843 raft_consensus.cc:3037] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:01:10.011673 4572 consensus_queue.cc:237] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } }
I20240503 08:01:10.012497 3843 raft_consensus.cc:2454] T 6dbb6ccaa8d144c48404a95066a11ab1 P e01f09ad924f45c699fb12e45a597ac8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a08972a8b584403aaf1a7fb51b775a79 in term 2.
I20240503 08:01:10.013358 3661 catalog_manager.cc:5484] T 6dbb6ccaa8d144c48404a95066a11ab1 P a08972a8b584403aaf1a7fb51b775a79 reported cstate change: term changed from 1 to 2, leader changed from e01f09ad924f45c699fb12e45a597ac8 (127.0.99.2) to a08972a8b584403aaf1a7fb51b775a79 (127.0.99.6). New cstate: current_term: 2 leader_uuid: "a08972a8b584403aaf1a7fb51b775a79" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "44f0a2c3ee394d70ad7145e6b04ad926" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32789 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f09ad924f45c699fb12e45a597ac8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08972a8b584403aaf1a7fb51b775a79" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43249 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be55fde3594e417faf23e55a23944033" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34743 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:01:10.032667 4707 ts_tablet_manager.cc:902] T ade6e49ee7304d79b81c03cd034a815c P f8e707d0d0eb4812b9e22be35d86a152: Initiating tablet copy from peer 44f0a2c3ee394d70ad7145e6b04ad926 (127.0.99.1:32789)
F20240503 08:01:10.032891 4707 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-06177450ba6d45f48d6c3cf937737158.queue_time_us, tablet-download-pool-06177450ba6d45f48d6c3cf937737158.queue_time_us) (tablet-download-pool-06177450ba6d45f48d6c3cf937737158.run_cpu_time_us, tablet-download-pool-06177450ba6d45f48d6c3cf937737158.run_cpu_time_us) (tablet-download-pool-06177450ba6d45f48d6c3cf937737158.run_wall_time_us, tablet-download-pool-06177450ba6d45f48d6c3cf937737158.run_wall_time_us) (tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.queue_time_us, tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.queue_time_us) (tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.run_cpu_time_us, tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.run_cpu_time_us) (tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.run_wall_time_us, tablet-download-pool-1f4b086cb6a9411eac66da6d2506e590.run_wall_time_us) (tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.queue_time_us, tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.queue_time_us) (tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.run_cpu_time_us, tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.run_cpu_time_us) (tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.run_wall_time_us, tablet-download-pool-20be03fba1ee443dba1709cbc69707bd.run_wall_time_us) (tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.queue_time_us, tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.queue_time_us) (tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.run_cpu_time_us, tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.run_cpu_time_us) (tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.run_wall_time_us, tablet-download-pool-26e0121d8d964328ae012d07b34c26e7.run_wall_time_us) (tablet-download-pool-5899a1db66d148be808923d5b15aa34f.queue_time_us, tablet-download-pool-5899a1db66d148be808923d5b15aa34f.queue_time_us) (tablet-download-pool-5899a1db66d148be808923d5b15aa34f.run_cpu_time_us, tablet-download-pool-5899a1db66d148be808923d5b15aa34f.run_cpu_time_us) (tablet-download-pool-5899a1db66d148be808923d5b15aa34f.run_wall_time_us, tablet-download-pool-5899a1db66d148be808923d5b15aa34f.run_wall_time_us) (tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.queue_time_us, tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.queue_time_us) (tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.run_cpu_time_us, tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.run_cpu_time_us) (tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.run_wall_time_us, tablet-download-pool-6dbb6ccaa8d144c48404a95066a11ab1.run_wall_time_us) (tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.queue_time_us, tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.queue_time_us) (tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.run_cpu_time_us, tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.run_cpu_time_us) (tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.run_wall_time_us, tablet-download-pool-81111e10c6c44e3182ff19e2001a824f.run_wall_time_us) (tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.queue_time_us, tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.queue_time_us) (tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.run_cpu_time_us, tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.run_cpu_time_us) (tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.run_wall_time_us, tablet-download-pool-88348c5624c94fbcb125f112e65bdb34.run_wall_time_us) (tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.queue_time_us, tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.queue_time_us) (tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.run_cpu_time_us, tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.run_cpu_time_us) (tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.run_wall_time_us, tablet-download-pool-9811bcfc2e9f495ca6115718712ac4ff.run_wall_time_us) (tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.queue_time_us, tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.queue_time_us) (tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.run_cpu_time_us, tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.run_cpu_time_us) (tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.run_wall_time_us, tablet-download-pool-a8ecce677bc4429bbab195d09e822bc7.run_wall_time_us) (tablet-download-pool-ade6e49ee7304d79b81c03cd034a815c.queue_time_us, tablet-download-pool-ade6e49ee7304d79b81c03cd034a815c.queue_time_us) (tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.queue_time_us, tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.queue_time_us) (tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.run_cpu_time_us, tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.run_cpu_time_us) (tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.run_wall_time_us, tablet-download-pool-c59c68fd5da24311a940047a49eb36ec.run_wall_time_us) (tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.queue_time_us, tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.queue_time_us) (tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.run_cpu_time_us, tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.run_cpu_time_us) (tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.run_wall_time_us, tablet-download-pool-cf6b5bb3a0ab4bbc9314b9cd3634d373.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714723270 (unix time) try "date -d @1714723270" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7fe1ba139700) from PID 396; stack trace: ***
@ 0x7fe293431040 (unknown) at ??:0
@ 0x7fe293430fb7 gsignal at ??:0
@ 0x7fe293432921 abort at ??:0
@ 0x7fe294135dcd google::LogMessage::Fail() at ??:0
@ 0x7fe294139b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe2941357cc google::LogMessage::Flush() at ??:0
@ 0x7fe294136f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe294c8d044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe294c71f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe294c6fefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe295bec77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe295bf5a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe295c7676f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe295c8a4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe295c75226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fe295c84c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe2961df3cc std::function<>::operator()() at ??:0
@ 0x7fe294c74e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe294c756ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe294c7703b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe2961df3cc std::function<>::operator()() at ??:0
@ 0x7fe294c64807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2914666db start_thread at ??:0
@ 0x7fe29351371f clone at ??:0