[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 07:58:33.805178 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:39889,127.0.99.61:39117,127.0.99.60:39957
I20240501 07:58:33.806146 396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.806699 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.814024 401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.814494 404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.814980 402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.815207 396 server_base.cc:1034] running on GCE node
I20240501 07:58:33.815797 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:33.815914 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:33.815941 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550313815939 us; error 0 us; skew 500 ppm
I20240501 07:58:33.821992 396 webserver.cc:460] Webserver started at http://127.0.99.62:43725/ using document root <none> and password file <none>
I20240501 07:58:33.822551 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.822607 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.822839 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.824549 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "d7e0420c472548638969fb80e9bf06fa"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.827948 396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 07:58:33.829942 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.831012 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:33.831113 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "d7e0420c472548638969fb80e9bf06fa"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.831203 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.849992 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.850617 396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.850807 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.857133 461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:39889 every 8 connection(s)
I20240501 07:58:33.857141 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:39889
I20240501 07:58:33.858242 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.859719 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.860208 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.860407 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 07:58:33.860678 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.861111 396 server_base.cc:1034] running on GCE node
I20240501 07:58:33.861249 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:33.861284 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:33.861302 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550313861302 us; error 0 us; skew 500 ppm
I20240501 07:58:33.862108 396 webserver.cc:460] Webserver started at http://127.0.99.61:37911/ using document root <none> and password file <none>
I20240501 07:58:33.862232 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.862272 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.862339 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.862679 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root/instance:
uuid: "e19467211da140afb0afc40b2af220f7"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.864192 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:33.864773 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.865005 472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.865218 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:33.865275 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
uuid: "e19467211da140afb0afc40b2af220f7"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.865329 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.872781 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.873142 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.874086 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:33.874773 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.61:39117: connect: Connection refused (error 111)
W20240501 07:58:33.875524 462 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:39117: Network error: Client connection negotiation failed: client connection to 127.0.99.61:39117: connect: Connection refused (error 111)
I20240501 07:58:33.878021 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:39117
I20240501 07:58:33.878046 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:39117 every 8 connection(s)
I20240501 07:58:33.878602 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:33.878742 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 07:58:33.879925 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.880244 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:33.880421 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.880780 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.881199 396 server_base.cc:1034] running on GCE node
I20240501 07:58:33.881354 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:33.881388 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:33.881407 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550313881407 us; error 0 us; skew 500 ppm
I20240501 07:58:33.882215 396 webserver.cc:460] Webserver started at http://127.0.99.60:41499/ using document root <none> and password file <none>
I20240501 07:58:33.882329 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.882371 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.882433 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.882781 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root/instance:
uuid: "92d71916164043128ad45c913663e77a"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.884133 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.884109 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.885221 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.885452 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.885511 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
uuid: "92d71916164043128ad45c913663e77a"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.885568 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.887902 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:33.888939 528 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:39957: Network error: Client connection negotiation failed: client connection to 127.0.99.60:39957: connect: Connection refused (error 111)
I20240501 07:58:33.901587 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.901937 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.906240 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:39957
I20240501 07:58:33.906266 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:39957 every 8 connection(s)
I20240501 07:58:33.906610 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:58:33.907032 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.907986 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.911358 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.914317 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.919332 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: Bootstrap starting.
I20240501 07:58:33.921679 594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.922477 594 log.cc:824] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:33.923875 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: No bootstrap required, opened a new log
I20240501 07:58:33.924705 462 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } attempt: 1
I20240501 07:58:33.926764 594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.926904 594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.926987 594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d71916164043128ad45c913663e77a, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.927392 594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.927795 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:33.927945 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:33.928459 594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:33.928536 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:33.932988 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: Bootstrap starting.
W20240501 07:58:33.933485 610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:33.933557 610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:33.933898 462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.934844 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: No bootstrap required, opened a new log
I20240501 07:58:33.935256 462 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.935348 462 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.935369 462 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e0420c472548638969fb80e9bf06fa, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.935468 462 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.935751 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:33.935840 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:33.935904 462 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:33.937410 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 07:58:33.937889 624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:33.937943 624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:33.941072 528 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } attempt: 1
I20240501 07:58:33.945266 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: Bootstrap starting.
I20240501 07:58:33.946015 528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.946939 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: No bootstrap required, opened a new log
I20240501 07:58:33.947274 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.947352 528 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.947372 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19467211da140afb0afc40b2af220f7, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.947485 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.947928 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:33.947932 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:33.948052 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:33.949563 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240501 07:58:33.950019 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:33.950073 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:33.950141 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.952227 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.952809 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.953265 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.953694 396 server_base.cc:1034] running on GCE node
I20240501 07:58:33.953836 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:33.953879 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:33.953907 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550313953906 us; error 0 us; skew 500 ppm
I20240501 07:58:33.954835 396 webserver.cc:460] Webserver started at http://127.0.99.1:45995/ using document root <none> and password file <none>
I20240501 07:58:33.954988 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.955035 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.955103 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.955489 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.957082 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:33.958003 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.958233 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.958289 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.958348 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.970170 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.970845 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.971230 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:33.971971 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:33.972023 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.972071 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:33.972097 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.978381 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:36565
I20240501 07:58:33.978827 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:36565 every 8 connection(s)
I20240501 07:58:33.979513 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.981302 714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.981761 717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.982928 396 server_base.cc:1034] running on GCE node
I20240501 07:58:33.985168 613 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.985348 613 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
W20240501 07:58:33.985867 715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.986948 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:33.986997 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:33.987017 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550313987017 us; error 0 us; skew 500 ppm
I20240501 07:58:33.987030 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7e0420c472548638969fb80e9bf06fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d71916164043128ad45c913663e77a" is_pre_election: true
I20240501 07:58:33.987217 568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7e0420c472548638969fb80e9bf06fa in term 0.
I20240501 07:58:33.987553 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7e0420c472548638969fb80e9bf06fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19467211da140afb0afc40b2af220f7" is_pre_election: true
I20240501 07:58:33.987655 502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7e0420c472548638969fb80e9bf06fa in term 0.
I20240501 07:58:33.988047 712 heartbeater.cc:347] Connected to a master server at 127.0.99.61:39117
I20240501 07:58:33.988209 712 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.988361 396 webserver.cc:460] Webserver started at http://127.0.99.2:46605/ using document root <none> and password file <none>
I20240501 07:58:33.988490 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.988530 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.988652 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.988786 712 heartbeater.cc:511] Master 127.0.99.61:39117 requested a full tablet report, sending...
I20240501 07:58:33.989073 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "130c141a1bdf43deb31de32176314d2b"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.988039 411 leader_election.cc:302] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7e0420c472548638969fb80e9bf06fa, e19467211da140afb0afc40b2af220f7; no voters:
I20240501 07:58:33.990264 486 ts_manager.cc:194] Registered new tserver with Master: b2ddc28e1e5c4b91b3c1804c0ba950e1 (127.0.99.1:36565)
I20240501 07:58:33.990391 613 leader_election.cc:288] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e19467211da140afb0afc40b2af220f7 (127.0.99.61:39117), 92d71916164043128ad45c913663e77a (127.0.99.60:39957)
I20240501 07:58:33.990656 613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.990712 613 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.990749 613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.991715 613 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.992038 613 leader_election.cc:288] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [CANDIDATE]: Term 1 election: Requested vote from peers e19467211da140afb0afc40b2af220f7 (127.0.99.61:39117), 92d71916164043128ad45c913663e77a (127.0.99.60:39957)
I20240501 07:58:33.993263 711 heartbeater.cc:347] Connected to a master server at 127.0.99.60:39957
I20240501 07:58:33.993336 711 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.993535 711 heartbeater.cc:511] Master 127.0.99.60:39957 requested a full tablet report, sending...
I20240501 07:58:33.993742 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7e0420c472548638969fb80e9bf06fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d71916164043128ad45c913663e77a"
I20240501 07:58:33.993840 568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.994729 568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7e0420c472548638969fb80e9bf06fa in term 1.
I20240501 07:58:33.995062 558 ts_manager.cc:194] Registered new tserver with Master: b2ddc28e1e5c4b91b3c1804c0ba950e1 (127.0.99.1:36565)
I20240501 07:58:33.995182 396 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:33.995321 410 leader_election.cc:302] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d71916164043128ad45c913663e77a, d7e0420c472548638969fb80e9bf06fa; no voters:
I20240501 07:58:33.995496 613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.995579 613 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 1 LEADER]: Becoming Leader. State: Replica: d7e0420c472548638969fb80e9bf06fa, State: Running, Role: LEADER
I20240501 07:58:33.995826 613 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } }
I20240501 07:58:33.995955 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7e0420c472548638969fb80e9bf06fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19467211da140afb0afc40b2af220f7"
I20240501 07:58:33.996053 502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.996907 502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7e0420c472548638969fb80e9bf06fa in term 1.
I20240501 07:58:33.997380 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e0420c472548638969fb80e9bf06fa. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:33.997526 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:33.997599 710 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39889
I20240501 07:58:33.997812 710 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.998055 710 heartbeater.cc:511] Master 127.0.99.62:39889 requested a full tablet report, sending...
I20240501 07:58:33.998586 426 ts_manager.cc:194] Registered new tserver with Master: b2ddc28e1e5c4b91b3c1804c0ba950e1 (127.0.99.1:36565)
I20240501 07:58:33.998833 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.999074 396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240501 07:58:33.999135 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "130c141a1bdf43deb31de32176314d2b"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-rp4l"
I20240501 07:58:33.999193 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.999845 728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:34.000847 728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:34.005983 568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 1 FOLLOWER]: Refusing update from remote peer d7e0420c472548638969fb80e9bf06fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.006187 502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 1 FOLLOWER]: Refusing update from remote peer d7e0420c472548638969fb80e9bf06fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.006528 724 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.008018 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e0420c472548638969fb80e9bf06fa. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.008105 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:34.010028 613 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.011080 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e0420c472548638969fb80e9bf06fa. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.011171 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:34.011960 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.012039 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:34.012720 734 mvcc.cc:205] Tried to move back new op lower bound from 7022798086161358848 to 7022798086129893376. Current Snapshot: MvccSnapshot[applied={T|T < 7022798086161358848}]
I20240501 07:58:34.013038 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.013113 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:34.013842 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.013922 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:34.014565 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7e0420c472548638969fb80e9bf06fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7e0420c472548638969fb80e9bf06fa" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39889 } } peers { permanent_uuid: "e19467211da140afb0afc40b2af220f7" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 39117 } } peers { permanent_uuid: "92d71916164043128ad45c913663e77a" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 39957 } } }
I20240501 07:58:34.014662 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:34.016929 728 catalog_manager.cc:1347] Generated new cluster ID: b8c5b7d9ad0e41d8b2e1d036df349616
I20240501 07:58:34.016988 728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:34.024542 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:34.024984 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:34.025269 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:34.025820 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:34.025941 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.026089 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:34.026154 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.033130 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:37521
I20240501 07:58:34.034173 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:34.036487 800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:37521 every 8 connection(s)
W20240501 07:58:34.039206 806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:34.039829 801 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39889
I20240501 07:58:34.039907 801 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.039994 802 heartbeater.cc:347] Connected to a master server at 127.0.99.60:39957
I20240501 07:58:34.040061 802 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.040127 801 heartbeater.cc:511] Master 127.0.99.62:39889 requested a full tablet report, sending...
I20240501 07:58:34.040225 802 heartbeater.cc:511] Master 127.0.99.60:39957 requested a full tablet report, sending...
I20240501 07:58:34.040756 426 ts_manager.cc:194] Registered new tserver with Master: 130c141a1bdf43deb31de32176314d2b (127.0.99.2:37521)
I20240501 07:58:34.040788 558 ts_manager.cc:194] Registered new tserver with Master: 130c141a1bdf43deb31de32176314d2b (127.0.99.2:37521)
W20240501 07:58:34.041227 809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.041517 807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:34.044411 728 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:34.045888 728 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:34.046732 803 heartbeater.cc:347] Connected to a master server at 127.0.99.61:39117
I20240501 07:58:34.046808 803 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.046978 803 heartbeater.cc:511] Master 127.0.99.61:39117 requested a full tablet report, sending...
I20240501 07:58:34.047600 396 server_base.cc:1034] running on GCE node
I20240501 07:58:34.047667 486 ts_manager.cc:194] Registered new tserver with Master: 130c141a1bdf43deb31de32176314d2b (127.0.99.2:37521)
I20240501 07:58:34.047796 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:34.047829 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:34.047847 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550314047846 us; error 0 us; skew 500 ppm
I20240501 07:58:34.048794 396 webserver.cc:460] Webserver started at http://127.0.99.3:39645/ using document root <none> and password file <none>
I20240501 07:58:34.048936 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:34.048985 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:34.049080 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:34.049468 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "d66683581ea34e6c9b819e588689e0b6"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-rp4l"
I20240501 07:58:34.051074 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.052037 816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.052258 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.052317 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "d66683581ea34e6c9b819e588689e0b6"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-rp4l"
I20240501 07:58:34.052376 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:34.067967 728 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: Generated new TSK 0
I20240501 07:58:34.068465 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:34.068873 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:34.068884 728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:34.069212 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:34.069693 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:34.069758 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.069806 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:34.069833 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.075678 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:36683
I20240501 07:58:34.075731 879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:36683 every 8 connection(s)
I20240501 07:58:34.081228 880 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39889
I20240501 07:58:34.081338 880 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.081563 880 heartbeater.cc:511] Master 127.0.99.62:39889 requested a full tablet report, sending...
I20240501 07:58:34.081667 881 heartbeater.cc:347] Connected to a master server at 127.0.99.60:39957
I20240501 07:58:34.081751 881 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.081918 881 heartbeater.cc:511] Master 127.0.99.60:39957 requested a full tablet report, sending...
I20240501 07:58:34.082121 426 ts_manager.cc:194] Registered new tserver with Master: d66683581ea34e6c9b819e588689e0b6 (127.0.99.3:36683)
I20240501 07:58:34.082504 558 ts_manager.cc:194] Registered new tserver with Master: d66683581ea34e6c9b819e588689e0b6 (127.0.99.3:36683)
I20240501 07:58:34.082985 882 heartbeater.cc:347] Connected to a master server at 127.0.99.61:39117
I20240501 07:58:34.083062 882 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.083230 882 heartbeater.cc:511] Master 127.0.99.61:39117 requested a full tablet report, sending...
I20240501 07:58:34.083544 426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46210
I20240501 07:58:34.083772 486 ts_manager.cc:194] Registered new tserver with Master: d66683581ea34e6c9b819e588689e0b6 (127.0.99.3:36683)
I20240501 07:58:34.083871 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007706576s
I20240501 07:58:34.934808 610 catalog_manager.cc:1259] Loaded cluster ID: b8c5b7d9ad0e41d8b2e1d036df349616
I20240501 07:58:34.934891 610 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: loading cluster ID for follower catalog manager: success
I20240501 07:58:34.936259 610 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: acquiring CA information for follower catalog manager: success
I20240501 07:58:34.936939 610 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 07:58:34.950775 637 catalog_manager.cc:1259] Loaded cluster ID: b8c5b7d9ad0e41d8b2e1d036df349616
I20240501 07:58:34.950851 637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: loading cluster ID for follower catalog manager: success
I20240501 07:58:34.951979 637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: acquiring CA information for follower catalog manager: success
I20240501 07:58:34.952467 637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 07:58:35.000120 426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46190
I20240501 07:58:35.042702 426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46202
I20240501 07:58:35.084883 880 heartbeater.cc:503] Master 127.0.99.62:39889 was elected leader, sending a full tablet report...
I20240501 07:58:35.937520 623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:36.000998 710 heartbeater.cc:503] Master 127.0.99.62:39889 was elected leader, sending a full tablet report...
I20240501 07:58:36.043462 801 heartbeater.cc:503] Master 127.0.99.62:39889 was elected leader, sending a full tablet report...
I20240501 07:58:36.135061 396 test_util.cc:268] Using random seed: -628428144
I20240501 07:58:36.146795 426 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46218:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 07:58:36.163180 844 tablet_service.cc:1467] Processing CreateTablet for tablet 31b11eb695354508b882d4b0e8ff87d8 (DEFAULT_TABLE table=test-workload [id=0fb233e2291340f1905840868d38eeb8]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 07:58:36.163653 844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b11eb695354508b882d4b0e8ff87d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.163650 765 tablet_service.cc:1467] Processing CreateTablet for tablet 640f45576e97472c99cfac32ea3f75e1 (DEFAULT_TABLE table=test-workload [id=0fb233e2291340f1905840868d38eeb8]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 07:58:36.163951 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640f45576e97472c99cfac32ea3f75e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.165930 673 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc5b9b45b434803928a1438a37525a9 (DEFAULT_TABLE table=test-workload [id=0fb233e2291340f1905840868d38eeb8]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 07:58:36.165944 672 tablet_service.cc:1467] Processing CreateTablet for tablet 8dffcc3bbbf6402fbfe3b2c2793586e9 (DEFAULT_TABLE table=test-workload [id=0fb233e2291340f1905840868d38eeb8]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 07:58:36.166198 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc5b9b45b434803928a1438a37525a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.166301 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dffcc3bbbf6402fbfe3b2c2793586e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.166277 916 tablet_bootstrap.cc:492] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: Bootstrap starting.
I20240501 07:58:36.167218 916 tablet_bootstrap.cc:654] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.168207 916 tablet_bootstrap.cc:492] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: No bootstrap required, opened a new log
I20240501 07:58:36.168304 916 ts_tablet_manager.cc:1367] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:36.168335 915 tablet_bootstrap.cc:492] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: Bootstrap starting.
I20240501 07:58:36.168684 916 raft_consensus.cc:348] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130c141a1bdf43deb31de32176314d2b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 37521 } }
I20240501 07:58:36.168780 916 raft_consensus.cc:374] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.168809 916 raft_consensus.cc:729] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130c141a1bdf43deb31de32176314d2b, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.168921 916 consensus_queue.cc:260] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130c141a1bdf43deb31de32176314d2b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 37521 } }
I20240501 07:58:36.168998 916 raft_consensus.cc:388] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:36.169035 916 raft_consensus.cc:482] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:36.169073 916 raft_consensus.cc:3037] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.169117 915 tablet_bootstrap.cc:654] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.169759 916 raft_consensus.cc:504] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130c141a1bdf43deb31de32176314d2b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 37521 } }
I20240501 07:58:36.169874 916 leader_election.cc:302] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 130c141a1bdf43deb31de32176314d2b; no voters:
I20240501 07:58:36.169999 915 tablet_bootstrap.cc:492] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: No bootstrap required, opened a new log
I20240501 07:58:36.170042 916 leader_election.cc:288] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:36.170066 915 ts_tablet_manager.cc:1367] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:36.170096 920 raft_consensus.cc:2781] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.170233 920 raft_consensus.cc:686] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 1 LEADER]: Becoming Leader. State: Replica: 130c141a1bdf43deb31de32176314d2b, State: Running, Role: LEADER
I20240501 07:58:36.170315 916 ts_tablet_manager.cc:1398] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:36.170411 915 raft_consensus.cc:348] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66683581ea34e6c9b819e588689e0b6" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36683 } }
I20240501 07:58:36.170485 915 raft_consensus.cc:374] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.170513 915 raft_consensus.cc:729] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66683581ea34e6c9b819e588689e0b6, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.170341 920 consensus_queue.cc:237] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130c141a1bdf43deb31de32176314d2b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 37521 } }
I20240501 07:58:36.167861 918 tablet_bootstrap.cc:492] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Bootstrap starting.
I20240501 07:58:36.170611 915 consensus_queue.cc:260] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66683581ea34e6c9b819e588689e0b6" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36683 } }
I20240501 07:58:36.170706 915 raft_consensus.cc:388] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:36.170738 915 raft_consensus.cc:482] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:36.170769 915 raft_consensus.cc:3037] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.171350 918 tablet_bootstrap.cc:654] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.171479 915 raft_consensus.cc:504] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66683581ea34e6c9b819e588689e0b6" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36683 } }
I20240501 07:58:36.171595 915 leader_election.cc:302] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d66683581ea34e6c9b819e588689e0b6; no voters:
I20240501 07:58:36.171761 915 leader_election.cc:288] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:36.172042 915 ts_tablet_manager.cc:1398] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:36.172683 426 catalog_manager.cc:5484] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 130c141a1bdf43deb31de32176314d2b (127.0.99.2). New cstate: current_term: 1 leader_uuid: "130c141a1bdf43deb31de32176314d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130c141a1bdf43deb31de32176314d2b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 37521 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:36.173144 926 raft_consensus.cc:2781] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.173372 926 raft_consensus.cc:686] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 1 LEADER]: Becoming Leader. State: Replica: d66683581ea34e6c9b819e588689e0b6, State: Running, Role: LEADER
I20240501 07:58:36.173486 926 consensus_queue.cc:237] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66683581ea34e6c9b819e588689e0b6" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36683 } }
I20240501 07:58:36.178529 425 catalog_manager.cc:5484] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d66683581ea34e6c9b819e588689e0b6 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "d66683581ea34e6c9b819e588689e0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66683581ea34e6c9b819e588689e0b6" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36683 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:36.178881 918 tablet_bootstrap.cc:492] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: No bootstrap required, opened a new log
I20240501 07:58:36.178941 918 ts_tablet_manager.cc:1367] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 07:58:36.179229 918 raft_consensus.cc:348] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.179311 918 raft_consensus.cc:374] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.179342 918 raft_consensus.cc:729] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ddc28e1e5c4b91b3c1804c0ba950e1, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.179440 918 consensus_queue.cc:260] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.179524 918 raft_consensus.cc:388] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:36.179555 918 raft_consensus.cc:482] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:36.179590 918 raft_consensus.cc:3037] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.180333 918 raft_consensus.cc:504] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.180433 918 leader_election.cc:302] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ddc28e1e5c4b91b3c1804c0ba950e1; no voters:
I20240501 07:58:36.180568 918 leader_election.cc:288] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:36.180655 931 raft_consensus.cc:2781] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.180802 931 raft_consensus.cc:686] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 LEADER]: Becoming Leader. State: Replica: b2ddc28e1e5c4b91b3c1804c0ba950e1, State: Running, Role: LEADER
I20240501 07:58:36.180850 918 ts_tablet_manager.cc:1398] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:36.180904 931 consensus_queue.cc:237] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.181689 918 tablet_bootstrap.cc:492] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Bootstrap starting.
I20240501 07:58:36.183104 425 catalog_manager.cc:5484] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2ddc28e1e5c4b91b3c1804c0ba950e1 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:36.183226 918 tablet_bootstrap.cc:654] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.184177 918 tablet_bootstrap.cc:492] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: No bootstrap required, opened a new log
I20240501 07:58:36.184252 918 ts_tablet_manager.cc:1367] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:36.184604 918 raft_consensus.cc:348] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.184691 918 raft_consensus.cc:374] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.184722 918 raft_consensus.cc:729] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ddc28e1e5c4b91b3c1804c0ba950e1, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.184819 918 consensus_queue.cc:260] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.184887 918 raft_consensus.cc:388] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:36.184918 918 raft_consensus.cc:482] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:36.184952 918 raft_consensus.cc:3037] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.203351 918 raft_consensus.cc:504] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.203522 918 leader_election.cc:302] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ddc28e1e5c4b91b3c1804c0ba950e1; no voters:
I20240501 07:58:36.203630 918 leader_election.cc:288] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:36.203704 932 raft_consensus.cc:2781] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.203812 932 raft_consensus.cc:686] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 LEADER]: Becoming Leader. State: Replica: b2ddc28e1e5c4b91b3c1804c0ba950e1, State: Running, Role: LEADER
I20240501 07:58:36.203864 918 ts_tablet_manager.cc:1398] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: Time spent starting tablet: real 0.020s user 0.000s sys 0.000s
I20240501 07:58:36.203939 932 consensus_queue.cc:237] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } }
I20240501 07:58:36.205202 426 catalog_manager.cc:5484] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2ddc28e1e5c4b91b3c1804c0ba950e1 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ddc28e1e5c4b91b3c1804c0ba950e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36565 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:36.218365 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 07:58:36.222674 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:36.222932 396 tablet_replica.cc:330] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: stopping tablet replica
I20240501 07:58:36.223177 396 raft_consensus.cc:2229] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:36.223322 396 raft_consensus.cc:2258] T 8dffcc3bbbf6402fbfe3b2c2793586e9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.224336 396 tablet_replica.cc:330] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1: stopping tablet replica
I20240501 07:58:36.224418 396 raft_consensus.cc:2229] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:36.224520 396 raft_consensus.cc:2258] T 4cc5b9b45b434803928a1438a37525a9 P b2ddc28e1e5c4b91b3c1804c0ba950e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.238093 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:58:36.241573 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 07:58:36.244616 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:36.244812 396 tablet_replica.cc:330] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b: stopping tablet replica
I20240501 07:58:36.244925 396 raft_consensus.cc:2229] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:36.245046 396 raft_consensus.cc:2258] T 640f45576e97472c99cfac32ea3f75e1 P 130c141a1bdf43deb31de32176314d2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.257584 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 07:58:36.259459 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 07:58:36.263219 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:36.263411 396 tablet_replica.cc:330] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6: stopping tablet replica
I20240501 07:58:36.263523 396 raft_consensus.cc:2229] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:36.263620 396 raft_consensus.cc:2258] T 31b11eb695354508b882d4b0e8ff87d8 P d66683581ea34e6c9b819e588689e0b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.265941 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:58:36.267513 396 master.cc:537] Master@127.0.99.62:39889 shutting down...
I20240501 07:58:36.270357 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:36.270586 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.270666 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d7e0420c472548638969fb80e9bf06fa: stopping tablet replica
I20240501 07:58:36.289897 396 master.cc:552] Master@127.0.99.62:39889 shutdown complete.
I20240501 07:58:36.292593 396 master.cc:537] Master@127.0.99.61:39117 shutting down...
I20240501 07:58:36.295194 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:36.295336 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.295397 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P e19467211da140afb0afc40b2af220f7: stopping tablet replica
I20240501 07:58:36.297560 396 master.cc:552] Master@127.0.99.61:39117 shutdown complete.
I20240501 07:58:36.299464 396 master.cc:537] Master@127.0.99.60:39957 shutting down...
I20240501 07:58:36.302485 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:36.302605 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:36.302683 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92d71916164043128ad45c913663e77a: stopping tablet replica
I20240501 07:58:36.314818 396 master.cc:552] Master@127.0.99.60:39957 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2517 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 07:58:36.321444 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43815
I20240501 07:58:36.321751 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:36.323125 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.323721 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.324199 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.324663 396 server_base.cc:1034] running on GCE node
I20240501 07:58:36.324788 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:36.324822 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:36.324841 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550316324840 us; error 0 us; skew 500 ppm
I20240501 07:58:36.325594 396 webserver.cc:460] Webserver started at http://127.0.99.62:38767/ using document root <none> and password file <none>
I20240501 07:58:36.325718 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:36.325758 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:36.325826 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:36.326151 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "32bfcac3644f4ed4a610164bb6e1cb2d"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.327459 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 07:58:36.328222 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.328456 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:36.328512 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "32bfcac3644f4ed4a610164bb6e1cb2d"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.328563 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:36.333635 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:36.333876 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:36.337306 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43815
I20240501 07:58:36.339780 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43815 every 8 connection(s)
I20240501 07:58:36.340165 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.341787 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d: Bootstrap starting.
I20240501 07:58:36.342489 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.343423 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d: No bootstrap required, opened a new log
I20240501 07:58:36.343763 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER }
I20240501 07:58:36.343845 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.343875 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bfcac3644f4ed4a610164bb6e1cb2d, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.343994 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER }
I20240501 07:58:36.344055 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:36.344089 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:36.344131 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.344714 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER }
I20240501 07:58:36.344818 999 leader_election.cc:302] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32bfcac3644f4ed4a610164bb6e1cb2d; no voters:
I20240501 07:58:36.344976 999 leader_election.cc:288] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:36.345063 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.345261 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 1 LEADER]: Becoming Leader. State: Replica: 32bfcac3644f4ed4a610164bb6e1cb2d, State: Running, Role: LEADER
I20240501 07:58:36.345348 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:36.345391 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER }
I20240501 07:58:36.345793 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER } }
I20240501 07:58:36.345871 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32bfcac3644f4ed4a610164bb6e1cb2d. Latest consensus state: current_term: 1 leader_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bfcac3644f4ed4a610164bb6e1cb2d" member_type: VOTER } }
I20240501 07:58:36.345885 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:36.345978 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:36.346571 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:36.347420 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:36.347599 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:58:36.349128 1009 catalog_manager.cc:1347] Generated new cluster ID: 507d72d5f84340f782130cceecfbd15c
I20240501 07:58:36.349188 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:36.372365 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:36.372942 1009 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:36.378129 1009 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d: Generated new TSK 0
I20240501 07:58:36.378294 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:36.379644 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:36.381218 1020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.381681 1021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.382109 1023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.382556 396 server_base.cc:1034] running on GCE node
I20240501 07:58:36.382756 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:36.382802 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:36.382822 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550316382820 us; error 0 us; skew 500 ppm
I20240501 07:58:36.383710 396 webserver.cc:460] Webserver started at http://127.0.99.1:45539/ using document root <none> and password file <none>
I20240501 07:58:36.383848 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:36.383898 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:36.383972 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:36.384457 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "c1166e30323b4fc8833069b8064b2a48"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.385731 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:36.386510 1028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.386745 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.386807 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "c1166e30323b4fc8833069b8064b2a48"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.386871 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:36.396859 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:36.397133 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:36.397372 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:36.397769 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:36.397805 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.397841 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:36.397871 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.401589 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:46813
I20240501 07:58:36.401620 1091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:46813 every 8 connection(s)
I20240501 07:58:36.402277 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:36.403720 1096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.404129 1099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.404565 396 server_base.cc:1034] running on GCE node
W20240501 07:58:36.404937 1097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.405418 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:36.405476 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:36.405498 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550316405498 us; error 0 us; skew 500 ppm
I20240501 07:58:36.406291 396 webserver.cc:460] Webserver started at http://127.0.99.2:39711/ using document root <none> and password file <none>
I20240501 07:58:36.406419 1092 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43815
I20240501 07:58:36.406427 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:36.406507 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:36.406509 1092 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:36.406585 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:36.406749 1092 heartbeater.cc:511] Master 127.0.99.62:43815 requested a full tablet report, sending...
I20240501 07:58:36.406982 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.407351 963 ts_manager.cc:194] Registered new tserver with Master: c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813)
I20240501 07:58:36.408300 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55196
I20240501 07:58:36.408527 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:36.409430 1104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.409677 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.409732 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.409783 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:36.416934 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:36.417165 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:36.417363 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:36.417711 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:36.417743 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.417769 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:36.417786 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.421346 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:42315
I20240501 07:58:36.421367 1167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:42315 every 8 connection(s)
I20240501 07:58:36.422075 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:36.423746 1172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.423818 1171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.424201 1174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.424620 396 server_base.cc:1034] running on GCE node
I20240501 07:58:36.424768 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:36.424803 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:36.424822 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550316424822 us; error 0 us; skew 500 ppm
I20240501 07:58:36.425721 1168 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43815
I20240501 07:58:36.425734 396 webserver.cc:460] Webserver started at http://127.0.99.3:43113/ using document root <none> and password file <none>
I20240501 07:58:36.425808 1168 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:36.425884 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:36.425932 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:36.425995 1168 heartbeater.cc:511] Master 127.0.99.62:43815 requested a full tablet report, sending...
I20240501 07:58:36.426005 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:36.426430 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.426559 963 ts_manager.cc:194] Registered new tserver with Master: a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:36.427068 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55198
I20240501 07:58:36.427806 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.428602 1179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.428799 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.428861 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
format_stamp: "Formatted at 2024-05-01 07:58:36 on dist-test-slave-rp4l"
I20240501 07:58:36.428921 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:36.444586 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:36.444970 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:36.445261 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:36.445719 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:36.445757 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.445797 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:36.445847 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.450172 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:42571
I20240501 07:58:36.450203 1242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:42571 every 8 connection(s)
I20240501 07:58:36.454106 1243 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43815
I20240501 07:58:36.454195 1243 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:36.454368 1243 heartbeater.cc:511] Master 127.0.99.62:43815 requested a full tablet report, sending...
I20240501 07:58:36.454900 963 ts_manager.cc:194] Registered new tserver with Master: 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:36.455034 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004497597s
I20240501 07:58:36.455543 963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55212
I20240501 07:58:37.409189 1092 heartbeater.cc:503] Master 127.0.99.62:43815 was elected leader, sending a full tablet report...
I20240501 07:58:37.427815 1168 heartbeater.cc:503] Master 127.0.99.62:43815 was elected leader, sending a full tablet report...
I20240501 07:58:37.456254 1243 heartbeater.cc:503] Master 127.0.99.62:43815 was elected leader, sending a full tablet report...
I20240501 07:58:37.480804 396 test_util.cc:268] Using random seed: -627082399
I20240501 07:58:37.487941 963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55218:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 07:58:37.488449 963 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 07:58:37.498716 1207 tablet_service.cc:1467] Processing CreateTablet for tablet 7091110404b84ae2aa99420516d2a6aa (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 07:58:37.499027 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091110404b84ae2aa99420516d2a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.499030 1201 tablet_service.cc:1467] Processing CreateTablet for tablet de7a2d7d23034b559259da3b5f7b1895 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 07:58:37.499429 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a2d7d23034b559259da3b5f7b1895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.499552 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 7091110404b84ae2aa99420516d2a6aa (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 07:58:37.499771 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091110404b84ae2aa99420516d2a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.499876 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 86220bd92db245039986c7cba23449f9 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 07:58:37.500098 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86220bd92db245039986c7cba23449f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.501374 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 86220bd92db245039986c7cba23449f9 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 07:58:37.501427 1263 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.501590 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86220bd92db245039986c7cba23449f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.501994 1130 tablet_service.cc:1467] Processing CreateTablet for tablet a4528d1ca05f40f6b5146806827e271a (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 07:58:37.502202 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4528d1ca05f40f6b5146806827e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.502310 1263 tablet_bootstrap.cc:654] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.502873 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e7b1b60334dfca75e4b349d978593 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 07:58:37.503041 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e7b1b60334dfca75e4b349d978593. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.503871 1129 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4a611fd94b4fde8388e0138b85f146 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 07:58:37.504035 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4a611fd94b4fde8388e0138b85f146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.504518 1202 tablet_service.cc:1467] Processing CreateTablet for tablet 15552e261344443eafb674115e4e9a2f (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 07:58:37.504729 1202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15552e261344443eafb674115e4e9a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.504926 1125 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e7b1b60334dfca75e4b349d978593 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 07:58:37.505159 1125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e7b1b60334dfca75e4b349d978593. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.506245 1203 tablet_service.cc:1467] Processing CreateTablet for tablet 902e92d173674336b9a9ed1adb5009f6 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 07:58:37.506439 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e92d173674336b9a9ed1adb5009f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.506606 1126 tablet_service.cc:1467] Processing CreateTablet for tablet de7a2d7d23034b559259da3b5f7b1895 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 07:58:37.506844 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a2d7d23034b559259da3b5f7b1895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.506880 1205 tablet_service.cc:1467] Processing CreateTablet for tablet a4528d1ca05f40f6b5146806827e271a (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 07:58:37.507035 1127 tablet_service.cc:1467] Processing CreateTablet for tablet 15552e261344443eafb674115e4e9a2f (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 07:58:37.507267 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15552e261344443eafb674115e4e9a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.507320 1204 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4a611fd94b4fde8388e0138b85f146 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 07:58:37.506850 1128 tablet_service.cc:1467] Processing CreateTablet for tablet 902e92d173674336b9a9ed1adb5009f6 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 07:58:37.508721 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e92d173674336b9a9ed1adb5009f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.508898 1264 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.509809 1264 tablet_bootstrap.cc:654] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.507090 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4528d1ca05f40f6b5146806827e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.510383 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4a611fd94b4fde8388e0138b85f146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.511492 1264 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.511569 1264 ts_tablet_manager.cc:1367] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:37.511962 1264 raft_consensus.cc:348] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.512056 1264 raft_consensus.cc:374] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.512085 1264 raft_consensus.cc:729] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.512370 1264 consensus_queue.cc:260] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.512709 1056 tablet_service.cc:1467] Processing CreateTablet for tablet 7091110404b84ae2aa99420516d2a6aa (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 07:58:37.512924 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7091110404b84ae2aa99420516d2a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.513114 1264 ts_tablet_manager.cc:1398] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.513291 1264 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.514042 1264 tablet_bootstrap.cc:654] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.514686 1269 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.514843 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 86220bd92db245039986c7cba23449f9 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 07:58:37.515069 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86220bd92db245039986c7cba23449f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.515414 1269 tablet_bootstrap.cc:654] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.516067 1264 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.516158 1264 ts_tablet_manager.cc:1367] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:37.516306 1269 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.516381 1269 ts_tablet_manager.cc:1367] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.516534 1264 raft_consensus.cc:348] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.516633 1054 tablet_service.cc:1467] Processing CreateTablet for tablet a4528d1ca05f40f6b5146806827e271a (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 07:58:37.516741 1269 raft_consensus.cc:348] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.516824 1269 raft_consensus.cc:374] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.516790 1053 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4a611fd94b4fde8388e0138b85f146 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 07:58:37.516862 1269 raft_consensus.cc:729] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.516882 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4528d1ca05f40f6b5146806827e271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.517015 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4a611fd94b4fde8388e0138b85f146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.516978 1269 consensus_queue.cc:260] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.517341 1269 ts_tablet_manager.cc:1398] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.517498 1269 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.518488 1269 tablet_bootstrap.cc:654] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.518875 1052 tablet_service.cc:1467] Processing CreateTablet for tablet 902e92d173674336b9a9ed1adb5009f6 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 07:58:37.516631 1264 raft_consensus.cc:374] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.519207 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902e92d173674336b9a9ed1adb5009f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.519179 1264 raft_consensus.cc:729] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.519341 1264 consensus_queue.cc:260] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.519362 1050 tablet_service.cc:1467] Processing CreateTablet for tablet de7a2d7d23034b559259da3b5f7b1895 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 07:58:37.519562 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7a2d7d23034b559259da3b5f7b1895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.519578 1264 ts_tablet_manager.cc:1398] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:37.519719 1264 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.520511 1264 tablet_bootstrap.cc:654] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.521222 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2e7b1b60334dfca75e4b349d978593 (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 07:58:37.521414 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2e7b1b60334dfca75e4b349d978593. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.521411 1269 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.521531 1269 ts_tablet_manager.cc:1367] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:37.521719 1264 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.521800 1264 ts_tablet_manager.cc:1367] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.521924 1269 raft_consensus.cc:348] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.522019 1269 raft_consensus.cc:374] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.522056 1269 raft_consensus.cc:729] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.522162 1264 raft_consensus.cc:348] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.522168 1269 consensus_queue.cc:260] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.522264 1264 raft_consensus.cc:374] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.522294 1264 raft_consensus.cc:729] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.522385 1264 consensus_queue.cc:260] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.522424 1269 ts_tablet_manager.cc:1398] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:37.522615 1264 ts_tablet_manager.cc:1398] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.522820 1264 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.523092 1269 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.519186 1051 tablet_service.cc:1467] Processing CreateTablet for tablet 15552e261344443eafb674115e4e9a2f (DEFAULT_TABLE table=test-workload [id=e564334d075e47ad8272804104ffe4f9]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 07:58:37.523512 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15552e261344443eafb674115e4e9a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.523598 1264 tablet_bootstrap.cc:654] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.524082 1269 tablet_bootstrap.cc:654] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.524729 1264 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.524806 1264 ts_tablet_manager.cc:1367] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:37.525985 1269 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.526060 1269 ts_tablet_manager.cc:1367] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:37.526458 1263 tablet_bootstrap.cc:492] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.526543 1267 raft_consensus.cc:482] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.526552 1263 ts_tablet_manager.cc:1367] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240501 07:58:37.526643 1267 raft_consensus.cc:504] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.526878 1264 raft_consensus.cc:348] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.526966 1264 raft_consensus.cc:374] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.526997 1264 raft_consensus.cc:729] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.526985 1263 raft_consensus.cc:348] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.527056 1263 raft_consensus.cc:374] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.527087 1267 leader_election.cc:288] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.526423 1269 raft_consensus.cc:348] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.527195 1269 raft_consensus.cc:374] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.527236 1269 raft_consensus.cc:729] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.527082 1263 raft_consensus.cc:729] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.527354 1269 consensus_queue.cc:260] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.527388 1263 consensus_queue.cc:260] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.527597 1269 ts_tablet_manager.cc:1398] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:37.527702 1263 ts_tablet_manager.cc:1398] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.527770 1269 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.527842 1263 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.528585 1269 tablet_bootstrap.cc:654] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.528703 1263 tablet_bootstrap.cc:654] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.527113 1264 consensus_queue.cc:260] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.529480 1264 ts_tablet_manager.cc:1398] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 07:58:37.529630 1264 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.530406 1264 tablet_bootstrap.cc:654] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.530869 1263 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.530946 1263 ts_tablet_manager.cc:1367] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:37.531634 1263 raft_consensus.cc:348] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.531749 1263 raft_consensus.cc:374] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.531790 1263 raft_consensus.cc:729] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.531898 1263 consensus_queue.cc:260] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.532126 1263 ts_tablet_manager.cc:1398] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.531250 1269 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.532249 1269 ts_tablet_manager.cc:1367] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:37.532527 1263 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.532619 1269 raft_consensus.cc:348] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.532733 1269 raft_consensus.cc:374] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.532770 1269 raft_consensus.cc:729] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.532866 1269 consensus_queue.cc:260] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.533090 1269 ts_tablet_manager.cc:1398] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.533237 1269 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.534094 1263 tablet_bootstrap.cc:654] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.534101 1269 tablet_bootstrap.cc:654] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.535883 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.536003 1066 raft_consensus.cc:2454] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 0.
I20240501 07:58:37.536082 1269 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.536154 1269 ts_tablet_manager.cc:1367] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:37.536294 1107 leader_election.cc:302] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.536448 1267 raft_consensus.cc:2781] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.536511 1267 raft_consensus.cc:482] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.536550 1267 raft_consensus.cc:3037] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.536528 1269 raft_consensus.cc:348] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.536605 1269 raft_consensus.cc:374] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.536652 1269 raft_consensus.cc:729] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.536760 1269 consensus_queue.cc:260] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.537046 1269 ts_tablet_manager.cc:1398] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.537184 1269 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.537497 1267 raft_consensus.cc:504] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.537833 1267 leader_election.cc:288] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.537951 1269 tablet_bootstrap.cc:654] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.537981 1284 raft_consensus.cc:482] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.538136 1284 raft_consensus.cc:504] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.538364 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" is_pre_election: true
I20240501 07:58:37.538569 1284 leader_election.cc:288] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.538727 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:37.538870 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" is_pre_election: true
W20240501 07:58:37.539065 1108 leader_election.cc:341] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:37.539170 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
W20240501 07:58:37.539207 1108 leader_election.cc:341] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:37.539249 1066 raft_consensus.cc:3037] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.539260 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
W20240501 07:58:37.539299 1108 leader_election.cc:341] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:37.539356 1065 raft_consensus.cc:2454] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 0.
I20240501 07:58:37.539561 1107 leader_election.cc:302] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters: 6137c4720b3d48d88fc7113dffe9e3a9
I20240501 07:58:37.539866 1264 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.539930 1264 ts_tablet_manager.cc:1367] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 07:58:37.540031 1066 raft_consensus.cc:2454] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.540233 1264 raft_consensus.cc:348] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.540285 1107 leader_election.cc:302] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters: 6137c4720b3d48d88fc7113dffe9e3a9
I20240501 07:58:37.540320 1264 raft_consensus.cc:374] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.540343 1264 raft_consensus.cc:729] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.540417 1284 raft_consensus.cc:2781] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.540472 1284 raft_consensus.cc:482] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.540453 1264 consensus_queue.cc:260] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.540504 1284 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.540730 1286 raft_consensus.cc:2781] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.540779 1264 ts_tablet_manager.cc:1398] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:37.540925 1264 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.540910 1286 raft_consensus.cc:686] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Becoming Leader. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Running, Role: LEADER
I20240501 07:58:37.541054 1286 consensus_queue.cc:237] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.541234 1284 raft_consensus.cc:504] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.541739 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:37.541739 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.541854 1066 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:58:37.542035 1108 leader_election.cc:341] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:37.542121 1284 leader_election.cc:288] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.542529 1066 raft_consensus.cc:2454] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.542594 1269 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.542701 1269 ts_tablet_manager.cc:1367] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:37.541774 1264 tablet_bootstrap.cc:654] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.542832 962 catalog_manager.cc:5484] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2). New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:37.543077 1269 raft_consensus.cc:348] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.543164 1269 raft_consensus.cc:374] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.543195 1269 raft_consensus.cc:729] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.543288 1269 consensus_queue.cc:260] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.543422 1107 leader_election.cc:302] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters: 6137c4720b3d48d88fc7113dffe9e3a9
I20240501 07:58:37.543503 1269 ts_tablet_manager.cc:1398] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.543650 1269 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.544298 1269 tablet_bootstrap.cc:654] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.545265 1269 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.545342 1269 ts_tablet_manager.cc:1367] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:37.545696 1263 tablet_bootstrap.cc:492] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.545766 1263 ts_tablet_manager.cc:1367] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240501 07:58:37.545734 1269 raft_consensus.cc:348] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.545931 1269 raft_consensus.cc:374] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.545980 1269 raft_consensus.cc:729] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.546140 1269 consensus_queue.cc:260] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.546170 1263 raft_consensus.cc:348] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.546270 1263 raft_consensus.cc:374] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.546310 1263 raft_consensus.cc:729] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.546417 1263 consensus_queue.cc:260] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.546468 1269 ts_tablet_manager.cc:1398] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.546674 1263 ts_tablet_manager.cc:1398] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.546679 1269 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: Bootstrap starting.
I20240501 07:58:37.546842 1263 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.547048 1284 raft_consensus.cc:2781] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.547873 1284 raft_consensus.cc:686] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Becoming Leader. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Running, Role: LEADER
I20240501 07:58:37.547986 1284 consensus_queue.cc:237] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.547678 1263 tablet_bootstrap.cc:654] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.547581 1269 tablet_bootstrap.cc:654] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.550027 1264 tablet_bootstrap.cc:492] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.550104 1264 ts_tablet_manager.cc:1367] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 07:58:37.550493 1264 raft_consensus.cc:348] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.550577 1264 raft_consensus.cc:374] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.550613 1264 raft_consensus.cc:729] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.550737 1264 consensus_queue.cc:260] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.550947 1264 ts_tablet_manager.cc:1398] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.551075 1264 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.551537 1269 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: No bootstrap required, opened a new log
I20240501 07:58:37.551559 1263 tablet_bootstrap.cc:492] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.551607 1269 ts_tablet_manager.cc:1367] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:37.551614 1263 ts_tablet_manager.cc:1367] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:37.551882 1264 tablet_bootstrap.cc:654] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.551992 1269 raft_consensus.cc:348] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.552078 1269 raft_consensus.cc:374] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.552109 1269 raft_consensus.cc:729] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.552139 1263 raft_consensus.cc:348] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.552206 1269 consensus_queue.cc:260] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.552265 1263 raft_consensus.cc:374] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.552299 1263 raft_consensus.cc:729] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.552425 1269 ts_tablet_manager.cc:1398] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.552460 1263 consensus_queue.cc:260] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.552572 1267 raft_consensus.cc:482] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.552649 1267 raft_consensus.cc:504] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.552796 1263 ts_tablet_manager.cc:1398] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.552918 1267 leader_election.cc:288] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.553015 1263 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.553267 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e7b1b60334dfca75e4b349d978593" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.553356 1066 raft_consensus.cc:2454] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 0.
I20240501 07:58:37.553562 1107 leader_election.cc:302] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.554008 1267 raft_consensus.cc:2781] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.554054 1267 raft_consensus.cc:482] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.554085 1267 raft_consensus.cc:3037] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.554190 1263 tablet_bootstrap.cc:654] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.554991 1267 raft_consensus.cc:504] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.555258 1267 leader_election.cc:288] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.555549 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e7b1b60334dfca75e4b349d978593" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:37.555640 1215 raft_consensus.cc:3037] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.556449 1215 raft_consensus.cc:2454] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.556680 1108 leader_election.cc:302] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, a0d63cf6e0fa4163ab2a6fee244b242d; no voters:
I20240501 07:58:37.556972 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e7b1b60334dfca75e4b349d978593" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.557049 1066 raft_consensus.cc:3037] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.557754 1066 raft_consensus.cc:2454] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.558058 1267 raft_consensus.cc:2781] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.558123 1267 raft_consensus.cc:686] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Becoming Leader. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Running, Role: LEADER
I20240501 07:58:37.558216 1267 consensus_queue.cc:237] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.559374 1263 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.559442 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2e7b1b60334dfca75e4b349d978593" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" is_pre_election: true
I20240501 07:58:37.559475 1263 ts_tablet_manager.cc:1367] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:37.559556 1217 raft_consensus.cc:2362] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1. Re-sending same reply.
I20240501 07:58:37.559880 1263 raft_consensus.cc:348] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.559978 1263 raft_consensus.cc:374] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.560014 1263 raft_consensus.cc:729] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.560117 1263 consensus_queue.cc:260] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.560353 1263 ts_tablet_manager.cc:1398] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.560500 1263 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.561082 963 catalog_manager.cc:5484] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2). New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:37.561282 1264 tablet_bootstrap.cc:492] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.561359 1264 ts_tablet_manager.cc:1367] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:58:37.561406 1263 tablet_bootstrap.cc:654] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.561343 963 catalog_manager.cc:5484] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2). New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:37.561769 1264 raft_consensus.cc:348] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.561862 1264 raft_consensus.cc:374] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.561899 1264 raft_consensus.cc:729] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.561996 1264 consensus_queue.cc:260] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.562227 1264 ts_tablet_manager.cc:1398] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.562376 1264 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: Bootstrap starting.
I20240501 07:58:37.563220 1264 tablet_bootstrap.cc:654] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.564191 1264 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: No bootstrap required, opened a new log
I20240501 07:58:37.564272 1264 ts_tablet_manager.cc:1367] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.564307 1263 tablet_bootstrap.cc:492] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.564378 1263 ts_tablet_manager.cc:1367] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:37.564646 1263 raft_consensus.cc:348] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.564704 1263 raft_consensus.cc:374] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.564723 1263 raft_consensus.cc:729] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.564795 1263 consensus_queue.cc:260] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.564949 1263 ts_tablet_manager.cc:1398] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.564980 1264 raft_consensus.cc:348] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.565063 1264 raft_consensus.cc:374] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.565079 1263 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.565093 1264 raft_consensus.cc:729] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.565223 1264 consensus_queue.cc:260] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.565454 1264 ts_tablet_manager.cc:1398] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.565872 1263 tablet_bootstrap.cc:654] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.566817 1263 tablet_bootstrap.cc:492] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.566884 1263 ts_tablet_manager.cc:1367] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.567176 1263 raft_consensus.cc:348] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.567248 1263 raft_consensus.cc:374] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.567282 1263 raft_consensus.cc:729] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.567389 1263 consensus_queue.cc:260] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.567611 1263 ts_tablet_manager.cc:1398] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.567780 1263 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: Bootstrap starting.
I20240501 07:58:37.568568 1263 tablet_bootstrap.cc:654] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.569455 1263 tablet_bootstrap.cc:492] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: No bootstrap required, opened a new log
I20240501 07:58:37.569540 1263 ts_tablet_manager.cc:1367] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:37.569864 1263 raft_consensus.cc:348] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.569938 1263 raft_consensus.cc:374] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.569965 1263 raft_consensus.cc:729] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.570068 1263 consensus_queue.cc:260] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.570257 1263 ts_tablet_manager.cc:1398] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.624855 1271 raft_consensus.cc:482] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.624979 1271 raft_consensus.cc:504] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.625417 1271 leader_election.cc:288] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.628759 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091110404b84ae2aa99420516d2a6aa" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" is_pre_election: true
I20240501 07:58:37.628911 1217 raft_consensus.cc:2454] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 0.
I20240501 07:58:37.629248 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091110404b84ae2aa99420516d2a6aa" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:37.629261 1032 leader_election.cc:302] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.629390 1142 raft_consensus.cc:2454] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 0.
I20240501 07:58:37.629446 1271 raft_consensus.cc:2781] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.629495 1271 raft_consensus.cc:482] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.629537 1271 raft_consensus.cc:3037] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.630347 1271 raft_consensus.cc:504] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.630666 1271 leader_election.cc:288] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 1 election: Requested vote from peers 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.630928 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091110404b84ae2aa99420516d2a6aa" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:37.630937 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7091110404b84ae2aa99420516d2a6aa" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
I20240501 07:58:37.631019 1217 raft_consensus.cc:3037] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.631045 1142 raft_consensus.cc:3037] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.631711 1217 raft_consensus.cc:2454] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 1.
I20240501 07:58:37.631790 1142 raft_consensus.cc:2454] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 1.
I20240501 07:58:37.631933 1032 leader_election.cc:302] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.632074 1271 raft_consensus.cc:2781] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.632251 1271 raft_consensus.cc:686] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 LEADER]: Becoming Leader. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Running, Role: LEADER
I20240501 07:58:37.632376 1271 consensus_queue.cc:237] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } }
I20240501 07:58:37.633746 961 catalog_manager.cc:5484] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1166e30323b4fc8833069b8064b2a48 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "c1166e30323b4fc8833069b8064b2a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:37.644237 1284 raft_consensus.cc:482] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.644336 1284 raft_consensus.cc:504] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.644632 1284 leader_election.cc:288] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.644944 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e92d173674336b9a9ed1adb5009f6" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.644944 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e92d173674336b9a9ed1adb5009f6" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" is_pre_election: true
I20240501 07:58:37.645097 1217 raft_consensus.cc:2454] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 0.
I20240501 07:58:37.645115 1066 raft_consensus.cc:2454] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 0.
I20240501 07:58:37.645330 1107 leader_election.cc:302] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.645474 1284 raft_consensus.cc:2781] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.645527 1284 raft_consensus.cc:482] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.645551 1284 raft_consensus.cc:3037] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.646328 1284 raft_consensus.cc:504] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.646649 1284 leader_election.cc:288] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:37.646885 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e92d173674336b9a9ed1adb5009f6" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.646901 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902e92d173674336b9a9ed1adb5009f6" candidate_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:37.647006 1217 raft_consensus.cc:3037] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.647006 1066 raft_consensus.cc:3037] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.647827 1066 raft_consensus.cc:2454] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.647827 1217 raft_consensus.cc:2454] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d63cf6e0fa4163ab2a6fee244b242d in term 1.
I20240501 07:58:37.648072 1108 leader_election.cc:302] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, a0d63cf6e0fa4163ab2a6fee244b242d; no voters:
I20240501 07:58:37.648222 1284 raft_consensus.cc:2781] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.648298 1284 raft_consensus.cc:686] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Becoming Leader. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Running, Role: LEADER
I20240501 07:58:37.648413 1284 consensus_queue.cc:237] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.649729 961 catalog_manager.cc:5484] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2). New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:37.652985 1093 tablet.cc:2369] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:37.683563 1277 raft_consensus.cc:482] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.683717 1277 raft_consensus.cc:504] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.684231 1277 leader_election.cc:288] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.687335 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15552e261344443eafb674115e4e9a2f" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:37.687335 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15552e261344443eafb674115e4e9a2f" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.687502 1066 raft_consensus.cc:2454] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.687502 1142 raft_consensus.cc:2454] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.687762 1183 leader_election.cc:302] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, a0d63cf6e0fa4163ab2a6fee244b242d; no voters:
I20240501 07:58:37.687935 1277 raft_consensus.cc:2781] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.688007 1277 raft_consensus.cc:482] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.688035 1277 raft_consensus.cc:3037] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.688791 1277 raft_consensus.cc:504] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.689088 1277 leader_election.cc:288] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.689352 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15552e261344443eafb674115e4e9a2f" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
I20240501 07:58:37.689352 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15552e261344443eafb674115e4e9a2f" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.689482 1142 raft_consensus.cc:3037] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.689482 1066 raft_consensus.cc:3037] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.690289 1142 raft_consensus.cc:2454] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.690289 1066 raft_consensus.cc:2454] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.690562 1182 leader_election.cc:302] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.690754 1277 raft_consensus.cc:2781] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.690939 1277 raft_consensus.cc:686] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Running, Role: LEADER
I20240501 07:58:37.691061 1277 consensus_queue.cc:237] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.692464 961 catalog_manager.cc:5484] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:37.829162 1277 raft_consensus.cc:482] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.829319 1277 raft_consensus.cc:504] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.829723 1277 leader_election.cc:288] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.830101 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.830130 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:37.830240 1066 raft_consensus.cc:2379] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6137c4720b3d48d88fc7113dffe9e3a9 in current term 1: Already voted for candidate a0d63cf6e0fa4163ab2a6fee244b242d in this term.
I20240501 07:58:37.830543 1183 leader_election.cc:302] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9; no voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48
I20240501 07:58:37.830726 1277 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.837353 1308 raft_consensus.cc:482] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.837458 1308 raft_consensus.cc:504] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.838021 1308 leader_election.cc:288] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.838197 1308 raft_consensus.cc:482] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.838284 1308 raft_consensus.cc:504] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.838363 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4a611fd94b4fde8388e0138b85f146" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.838366 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4a611fd94b4fde8388e0138b85f146" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:37.838464 1066 raft_consensus.cc:2454] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.838512 1142 raft_consensus.cc:2454] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.838603 1308 leader_election.cc:288] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.838726 1182 leader_election.cc:302] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.838835 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a2d7d23034b559259da3b5f7b1895" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:37.838876 1308 raft_consensus.cc:2781] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.838913 1308 raft_consensus.cc:482] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.838927 1066 raft_consensus.cc:2454] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.838994 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a2d7d23034b559259da3b5f7b1895" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:37.839100 1142 raft_consensus.cc:2454] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 0.
I20240501 07:58:37.839161 1182 leader_election.cc:302] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.839453 1310 raft_consensus.cc:2781] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.838960 1308 raft_consensus.cc:3037] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.839499 1310 raft_consensus.cc:482] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.839540 1310 raft_consensus.cc:3037] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.845723 1277 raft_consensus.cc:2726] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:37.846514 1308 raft_consensus.cc:504] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.846649 1310 raft_consensus.cc:504] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.846938 1310 leader_election.cc:288] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.846936 1308 leader_election.cc:288] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Requested vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:37.847316 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a2d7d23034b559259da3b5f7b1895" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.847368 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4a611fd94b4fde8388e0138b85f146" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
I20240501 07:58:37.847410 1066 raft_consensus.cc:3037] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.847430 1142 raft_consensus.cc:3037] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.847448 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4a611fd94b4fde8388e0138b85f146" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
I20240501 07:58:37.847539 1065 raft_consensus.cc:3037] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.847412 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7a2d7d23034b559259da3b5f7b1895" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
I20240501 07:58:37.848327 1065 raft_consensus.cc:2454] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.848345 1141 raft_consensus.cc:3037] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.848212 1142 raft_consensus.cc:2454] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.848579 1182 leader_election.cc:302] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.848721 1308 raft_consensus.cc:2781] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.848806 1308 raft_consensus.cc:686] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Running, Role: LEADER
I20240501 07:58:37.848212 1066 raft_consensus.cc:2454] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.849135 1182 leader_election.cc:302] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:37.848937 1308 consensus_queue.cc:237] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.849274 1310 raft_consensus.cc:2781] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.849351 1310 raft_consensus.cc:686] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6137c4720b3d48d88fc7113dffe9e3a9, State: Running, Role: LEADER
I20240501 07:58:37.849457 1310 consensus_queue.cc:237] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:37.850804 961 catalog_manager.cc:5484] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:37.853428 961 catalog_manager.cc:5484] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:37.856905 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:37.858497 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:37.859094 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:37.859594 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:37.860059 396 server_base.cc:1034] running on GCE node
I20240501 07:58:37.860232 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:37.860276 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:37.860296 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550317860296 us; error 0 us; skew 500 ppm
I20240501 07:58:37.861174 396 webserver.cc:460] Webserver started at http://127.0.99.4:46079/ using document root <none> and password file <none>
I20240501 07:58:37.861315 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:37.861358 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:37.861426 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:37.861771 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-3-root/instance:
uuid: "c445cc4740b84f69a27ba7cadc49d181"
format_stamp: "Formatted at 2024-05-01 07:58:37 on dist-test-slave-rp4l"
I20240501 07:58:37.863077 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:37.863852 1321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:37.864056 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.864114 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-3-root
uuid: "c445cc4740b84f69a27ba7cadc49d181"
format_stamp: "Formatted at 2024-05-01 07:58:37 on dist-test-slave-rp4l"
I20240501 07:58:37.864172 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714550313795535-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:37.875762 1141 raft_consensus.cc:2454] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6137c4720b3d48d88fc7113dffe9e3a9 in term 1.
I20240501 07:58:37.877399 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:37.877723 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:37.878006 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:37.878415 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:37.878453 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:37.878491 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:37.878518 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:37.883374 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:43395
I20240501 07:58:37.883414 1384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:43395 every 8 connection(s)
I20240501 07:58:37.887830 1385 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43815
I20240501 07:58:37.887921 1385 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:37.888118 1385 heartbeater.cc:511] Master 127.0.99.62:43815 requested a full tablet report, sending...
I20240501 07:58:37.888672 961 ts_manager.cc:194] Registered new tserver with Master: c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4:43395)
I20240501 07:58:37.889161 961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55230
I20240501 07:58:37.985286 1284 consensus_queue.cc:1035] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.988430 1267 consensus_queue.cc:1035] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.004231 1284 consensus_queue.cc:1035] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.006701 1284 consensus_queue.cc:1035] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.008843 1310 raft_consensus.cc:482] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.008934 1310 raft_consensus.cc:504] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:38.009311 1310 leader_election.cc:288] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813), a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315)
I20240501 07:58:38.009490 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1166e30323b4fc8833069b8064b2a48" is_pre_election: true
I20240501 07:58:38.009625 1066 raft_consensus.cc:2379] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6137c4720b3d48d88fc7113dffe9e3a9 in current term 1: Already voted for candidate a0d63cf6e0fa4163ab2a6fee244b242d in this term.
I20240501 07:58:38.009689 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86220bd92db245039986c7cba23449f9" candidate_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" is_pre_election: true
I20240501 07:58:38.009949 1183 leader_election.cc:302] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9; no voters: a0d63cf6e0fa4163ab2a6fee244b242d, c1166e30323b4fc8833069b8064b2a48
I20240501 07:58:38.010079 1310 raft_consensus.cc:3037] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.018707 1271 consensus_queue.cc:1035] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.020992 1396 consensus_queue.cc:1035] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.028024 1284 consensus_queue.cc:1035] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.030472 1267 consensus_queue.cc:1035] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.051347 1310 raft_consensus.cc:2726] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:38.066450 1267 consensus_queue.cc:1035] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.069172 1267 consensus_queue.cc:1035] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.180258 1310 consensus_queue.cc:1035] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.182895 1308 consensus_queue.cc:1035] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.280357 1277 consensus_queue.cc:1035] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.283178 1277 consensus_queue.cc:1035] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.315526 1277 consensus_queue.cc:1035] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.317996 1277 consensus_queue.cc:1035] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.347216 1017 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:38.352429 1141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a"
dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
mode: GRACEFUL
new_leader_uuid: "c1166e30323b4fc8833069b8064b2a48"
from {username='slave'} at 127.0.0.1:37472
I20240501 07:58:38.352595 1141 raft_consensus.cc:595] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Received request to transfer leadership to TS c1166e30323b4fc8833069b8064b2a48
I20240501 07:58:38.352962 1017 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 07:58:38.353154 1017 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:38.490250 1267 raft_consensus.cc:979] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: : Instructing follower c1166e30323b4fc8833069b8064b2a48 to start an election
I20240501 07:58:38.490370 1267 raft_consensus.cc:1067] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Signalling peer c1166e30323b4fc8833069b8064b2a48 to start an election
I20240501 07:58:38.490921 1066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a"
dest_uuid: "c1166e30323b4fc8833069b8064b2a48"
from {username='slave'} at 127.0.0.1:55020
I20240501 07:58:38.491072 1066 raft_consensus.cc:482] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:38.491135 1066 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:38.574378 1066 raft_consensus.cc:504] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:38.574797 1066 leader_election.cc:288] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 2 election: Requested vote from peers a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2:42315), 6137c4720b3d48d88fc7113dffe9e3a9 (127.0.99.3:42571)
I20240501 07:58:38.575150 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6137c4720b3d48d88fc7113dffe9e3a9"
I20240501 07:58:38.575153 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4528d1ca05f40f6b5146806827e271a" candidate_uuid: "c1166e30323b4fc8833069b8064b2a48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d"
I20240501 07:58:38.575294 1217 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:38.575369 1141 raft_consensus.cc:3032] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:38.575415 1141 raft_consensus.cc:729] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a0d63cf6e0fa4163ab2a6fee244b242d, State: Running, Role: LEADER
I20240501 07:58:38.575565 1141 consensus_queue.cc:260] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:38.575723 1141 raft_consensus.cc:3037] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:38.576150 1217 raft_consensus.cc:2454] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 2.
I20240501 07:58:38.576421 1032 leader_election.cc:302] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6137c4720b3d48d88fc7113dffe9e3a9, c1166e30323b4fc8833069b8064b2a48; no voters:
I20240501 07:58:38.576565 1141 raft_consensus.cc:2454] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1166e30323b4fc8833069b8064b2a48 in term 2.
I20240501 07:58:38.576573 1396 raft_consensus.cc:2781] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:38.576769 1396 raft_consensus.cc:686] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 2 LEADER]: Becoming Leader. State: Replica: c1166e30323b4fc8833069b8064b2a48, State: Running, Role: LEADER
I20240501 07:58:38.576898 1396 consensus_queue.cc:237] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } }
I20240501 07:58:38.578203 963 catalog_manager.cc:5484] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 reported cstate change: term changed from 1 to 2, leader changed from a0d63cf6e0fa4163ab2a6fee244b242d (127.0.99.2) to c1166e30323b4fc8833069b8064b2a48 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "c1166e30323b4fc8833069b8064b2a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:38.889902 1385 heartbeater.cc:503] Master 127.0.99.62:43815 was elected leader, sending a full tablet report...
I20240501 07:58:38.963683 1217 raft_consensus.cc:1261] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 2 FOLLOWER]: Refusing update from remote peer c1166e30323b4fc8833069b8064b2a48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:58:38.964089 1415 consensus_queue.cc:1035] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:38.966694 1141 raft_consensus.cc:1261] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 2 FOLLOWER]: Refusing update from remote peer c1166e30323b4fc8833069b8064b2a48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:58:38.967038 1415 consensus_queue.cc:1035] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:40.353461 1017 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:40.354305 1017 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:40.354386 1017 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:42.354748 1017 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:42.355579 1017 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:42.355674 1017 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:43.360010 1141 consensus_queue.cc:237] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } }
I20240501 07:58:43.361469 1066 raft_consensus.cc:1261] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Refusing update from remote peer a0d63cf6e0fa4163ab2a6fee244b242d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.361719 1217 raft_consensus.cc:1261] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Refusing update from remote peer a0d63cf6e0fa4163ab2a6fee244b242d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.361959 1423 consensus_queue.cc:1035] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.363447 1419 consensus_queue.cc:1035] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:43.365463 1108 consensus_peers.cc:485] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d -> Peer c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4:43395): Couldn't send request to peer c445cc4740b84f69a27ba7cadc49d181. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f2e7b1b60334dfca75e4b349d978593. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:43.365350 1419 raft_consensus.cc:2932] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.365604 1066 raft_consensus.cc:2932] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.367460 1217 raft_consensus.cc:2932] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.367552 961 catalog_manager.cc:5484] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 reported cstate change: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } attrs { replace: true } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.372051 1066 consensus_queue.cc:237] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } }
I20240501 07:58:43.373358 1217 raft_consensus.cc:1261] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Refusing update from remote peer c1166e30323b4fc8833069b8064b2a48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.373440 1141 raft_consensus.cc:1261] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Refusing update from remote peer c1166e30323b4fc8833069b8064b2a48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.373678 1437 consensus_queue.cc:1035] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.373837 1415 consensus_queue.cc:1035] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.375298 1415 raft_consensus.cc:2932] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.375674 1141 raft_consensus.cc:2932] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.376015 1217 raft_consensus.cc:2932] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.377696 962 catalog_manager.cc:5484] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "c1166e30323b4fc8833069b8064b2a48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.377791 1141 consensus_queue.cc:237] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } }
W20240501 07:58:43.378717 1032 consensus_peers.cc:485] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 -> Peer c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4:43395): Couldn't send request to peer c445cc4740b84f69a27ba7cadc49d181. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7091110404b84ae2aa99420516d2a6aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:43.379026 1066 raft_consensus.cc:1261] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Refusing update from remote peer a0d63cf6e0fa4163ab2a6fee244b242d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.379290 1424 consensus_queue.cc:1035] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:43.379595 1108 consensus_peers.cc:485] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d -> Peer c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4:43395): Couldn't send request to peer c445cc4740b84f69a27ba7cadc49d181. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86220bd92db245039986c7cba23449f9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:43.379968 1215 raft_consensus.cc:1261] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Refusing update from remote peer a0d63cf6e0fa4163ab2a6fee244b242d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.380302 1424 consensus_queue.cc:1035] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.381302 1429 raft_consensus.cc:2932] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.382041 1217 raft_consensus.cc:2932] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.383366 961 catalog_manager.cc:5484] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d reported cstate change: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:43.383613 1066 raft_consensus.cc:2932] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.387643 1217 consensus_queue.cc:237] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } }
I20240501 07:58:43.388897 1066 raft_consensus.cc:1261] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Refusing update from remote peer 6137c4720b3d48d88fc7113dffe9e3a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.389076 1141 raft_consensus.cc:1261] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Refusing update from remote peer 6137c4720b3d48d88fc7113dffe9e3a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.389322 1447 consensus_queue.cc:1035] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.389642 1446 consensus_queue.cc:1035] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:43.390733 1447 raft_consensus.cc:2932] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.391366 1141 raft_consensus.cc:2932] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
W20240501 07:58:43.392292 1183 consensus_peers.cc:485] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 -> Peer c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4:43395): Couldn't send request to peer c445cc4740b84f69a27ba7cadc49d181. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15552e261344443eafb674115e4e9a2f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:43.392787 1066 raft_consensus.cc:2932] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } } }
I20240501 07:58:43.393267 963 catalog_manager.cc:5484] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 reported cstate change: config changed from index -1 to 2, NON_VOTER c445cc4740b84f69a27ba7cadc49d181 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c1166e30323b4fc8833069b8064b2a48" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d63cf6e0fa4163ab2a6fee244b242d" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6137c4720b3d48d88fc7113dffe9e3a9" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c445cc4740b84f69a27ba7cadc49d181" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:43.396554 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
W20240501 07:58:43.397609 949 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240501 07:58:43.398864 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 07:58:43.398924 1011 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 07:58:43.400138 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:43.400277 396 tablet_replica.cc:330] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.400403 396 raft_consensus.cc:2229] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.400537 396 raft_consensus.cc:2258] T 15552e261344443eafb674115e4e9a2f P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.401047 396 tablet_replica.cc:330] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.401180 396 raft_consensus.cc:2229] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.401278 396 raft_consensus.cc:2258] T 5f2e7b1b60334dfca75e4b349d978593 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.401683 396 tablet_replica.cc:330] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.401778 396 raft_consensus.cc:2229] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.401875 396 raft_consensus.cc:2258] T 86220bd92db245039986c7cba23449f9 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.402310 396 tablet_replica.cc:330] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.402408 396 raft_consensus.cc:2229] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.402603 396 raft_consensus.cc:2258] T 7091110404b84ae2aa99420516d2a6aa P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.403048 396 tablet_replica.cc:330] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.403136 396 raft_consensus.cc:2229] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.403221 396 raft_consensus.cc:2258] T 1b4a611fd94b4fde8388e0138b85f146 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.403519 396 tablet_replica.cc:330] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.403607 396 raft_consensus.cc:2229] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 2 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.403748 396 raft_consensus.cc:2258] T a4528d1ca05f40f6b5146806827e271a P c1166e30323b4fc8833069b8064b2a48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.404073 396 tablet_replica.cc:330] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.404140 396 raft_consensus.cc:2229] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.404219 396 raft_consensus.cc:2258] T 902e92d173674336b9a9ed1adb5009f6 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.404525 396 tablet_replica.cc:330] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48: stopping tablet replica
I20240501 07:58:43.404606 396 raft_consensus.cc:2229] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.404685 396 raft_consensus.cc:2258] T de7a2d7d23034b559259da3b5f7b1895 P c1166e30323b4fc8833069b8064b2a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.419004 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:58:43.421304 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
W20240501 07:58:43.423123 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 07:58:43.423179 1011 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 07:58:43.424170 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 07:58:43.424252 1011 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 07:58:43.426697 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:43.426847 396 tablet_replica.cc:330] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.426970 396 raft_consensus.cc:2229] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.427143 396 raft_consensus.cc:2258] T 902e92d173674336b9a9ed1adb5009f6 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.427554 396 tablet_replica.cc:330] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.427646 396 raft_consensus.cc:2229] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.427753 396 raft_consensus.cc:2258] T 15552e261344443eafb674115e4e9a2f P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.428197 396 tablet_replica.cc:330] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.428342 396 raft_consensus.cc:2229] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.428505 396 raft_consensus.cc:2258] T 86220bd92db245039986c7cba23449f9 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.428929 396 tablet_replica.cc:330] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.429013 396 raft_consensus.cc:2229] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.429112 396 raft_consensus.cc:2258] T 7091110404b84ae2aa99420516d2a6aa P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.429558 396 tablet_replica.cc:330] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.429646 396 raft_consensus.cc:2229] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.429738 396 raft_consensus.cc:2258] T 1b4a611fd94b4fde8388e0138b85f146 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.430042 396 tablet_replica.cc:330] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.430157 396 raft_consensus.cc:2229] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.430250 396 raft_consensus.cc:2258] T a4528d1ca05f40f6b5146806827e271a P a0d63cf6e0fa4163ab2a6fee244b242d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.430521 396 tablet_replica.cc:330] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.430598 396 raft_consensus.cc:2229] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.430822 396 raft_consensus.cc:2258] T 5f2e7b1b60334dfca75e4b349d978593 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.431197 396 tablet_replica.cc:330] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d: stopping tablet replica
I20240501 07:58:43.431270 396 raft_consensus.cc:2229] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.431352 396 raft_consensus.cc:2258] T de7a2d7d23034b559259da3b5f7b1895 P a0d63cf6e0fa4163ab2a6fee244b242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.445677 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
W20240501 07:58:43.447928 1182 consensus_peers.cc:485] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 -> Peer c1166e30323b4fc8833069b8064b2a48 (127.0.99.1:46813): Couldn't send request to peer c1166e30323b4fc8833069b8064b2a48. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.1:46813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:43.448781 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
W20240501 07:58:43.450013 1011 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 07:58:43.450151 1011 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 07:58:43.453120 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:43.453264 396 tablet_replica.cc:330] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.453366 396 raft_consensus.cc:2229] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.453464 396 raft_consensus.cc:2258] T a4528d1ca05f40f6b5146806827e271a P 6137c4720b3d48d88fc7113dffe9e3a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.453821 396 tablet_replica.cc:330] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.453900 396 raft_consensus.cc:2229] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.454035 396 raft_consensus.cc:2258] T 1b4a611fd94b4fde8388e0138b85f146 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.454316 396 tablet_replica.cc:330] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.454389 396 raft_consensus.cc:2229] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.454504 396 raft_consensus.cc:2258] T de7a2d7d23034b559259da3b5f7b1895 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.454792 396 tablet_replica.cc:330] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.454865 396 raft_consensus.cc:2229] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.454953 396 raft_consensus.cc:2258] T 7091110404b84ae2aa99420516d2a6aa P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.455310 396 tablet_replica.cc:330] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.455379 396 raft_consensus.cc:2229] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.455520 396 raft_consensus.cc:2258] T 15552e261344443eafb674115e4e9a2f P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.455901 396 tablet_replica.cc:330] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.455983 396 raft_consensus.cc:2229] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.456069 396 raft_consensus.cc:2258] T 86220bd92db245039986c7cba23449f9 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.456459 396 tablet_replica.cc:330] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.456526 396 raft_consensus.cc:2229] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.456598 396 raft_consensus.cc:2258] T 5f2e7b1b60334dfca75e4b349d978593 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.457028 396 tablet_replica.cc:330] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9: stopping tablet replica
I20240501 07:58:43.457093 396 raft_consensus.cc:2229] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.457157 396 raft_consensus.cc:2258] T 902e92d173674336b9a9ed1adb5009f6 P 6137c4720b3d48d88fc7113dffe9e3a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.470259 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:58:43.472695 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240501 07:58:43.476413 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:43.488054 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240501 07:58:43.489681 396 master.cc:537] Master@127.0.99.62:43815 shutting down...
I20240501 07:58:43.492419 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.492556 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.492614 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 32bfcac3644f4ed4a610164bb6e1cb2d: stopping tablet replica
I20240501 07:58:43.504823 396 master.cc:552] Master@127.0.99.62:43815 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7192 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 07:58:43.514070 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:39011,127.0.99.61:37973,127.0.99.60:37659
I20240501 07:58:43.514436 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:43.516599 1456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.517211 1454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.517355 1453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.517850 396 server_base.cc:1034] running on GCE node
I20240501 07:58:43.517978 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.518013 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.518025 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323518025 us; error 0 us; skew 500 ppm
I20240501 07:58:43.518816 396 webserver.cc:460] Webserver started at http://127.0.99.62:37187/ using document root <none> and password file <none>
I20240501 07:58:43.518950 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.518991 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.519069 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.519400 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "44f018f2612d4349b8041f62903ccf41"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.520641 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:43.521433 1461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.521674 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.521733 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "44f018f2612d4349b8041f62903ccf41"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.521795 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:43.529886 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.530202 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.534020 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:39011
I20240501 07:58:43.535202 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:43.538615 1513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:39011 every 8 connection(s)
I20240501 07:58:43.539091 1514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 07:58:43.539392 1516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.539816 1517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.540127 1514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.540306 396 server_base.cc:1034] running on GCE node
W20240501 07:58:43.540680 1519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.540922 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.540974 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.540992 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323540991 us; error 0 us; skew 500 ppm
I20240501 07:58:43.541939 396 webserver.cc:460] Webserver started at http://127.0.99.61:41783/ using document root <none> and password file <none>
I20240501 07:58:43.542073 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.542115 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.542174 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.542533 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root/instance:
uuid: "fd156b4cbcda4d2fb95b85fe24485255"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.545490 396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:43.546072 1514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.547848 1527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.548110 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:43.548162 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
uuid: "fd156b4cbcda4d2fb95b85fe24485255"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.548218 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 07:58:43.550657 1514 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:37973: Network error: Client connection negotiation failed: client connection to 127.0.99.61:37973: connect: Connection refused (error 111)
I20240501 07:58:43.556088 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.556461 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.560907 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:37973
I20240501 07:58:43.560941 1579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:37973 every 8 connection(s)
I20240501 07:58:43.561673 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:43.561837 1580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.563283 1580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:43.563493 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.563699 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.564006 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.564450 396 server_base.cc:1034] running on GCE node
I20240501 07:58:43.564677 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.564729 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.564750 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323564750 us; error 0 us; skew 500 ppm
I20240501 07:58:43.565757 396 webserver.cc:460] Webserver started at http://127.0.99.60:37927/ using document root <none> and password file <none>
I20240501 07:58:43.565905 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.565961 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.566028 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.566421 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root/instance:
uuid: "95d2a53816544ba7880a474cc48ccbcc"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.567895 1580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.568128 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:43.569167 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.569427 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:43.569530 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
uuid: "95d2a53816544ba7880a474cc48ccbcc"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.569633 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:43.575899 1580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:43.577020 1580 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:37659: Network error: Client connection negotiation failed: client connection to 127.0.99.60:37659: connect: Connection refused (error 111)
I20240501 07:58:43.583806 1514 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } attempt: 1
I20240501 07:58:43.585686 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.586021 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.587116 1514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } has no permanent_uuid. Determining permanent_uuid...
W20240501 07:58:43.588129 1514 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:37659: Network error: Client connection negotiation failed: client connection to 127.0.99.60:37659: connect: Connection refused (error 111)
I20240501 07:58:43.589994 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:37659
I20240501 07:58:43.590013 1645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:37659 every 8 connection(s)
I20240501 07:58:43.590365 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:58:43.590752 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.591665 1646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.594780 1646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.597751 1646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } has no permanent_uuid. Determining permanent_uuid...
I20240501 07:58:43.603134 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: Bootstrap starting.
I20240501 07:58:43.604034 1646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.604990 1646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: No bootstrap required, opened a new log
I20240501 07:58:43.605377 1646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.605481 1646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.605508 1646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d2a53816544ba7880a474cc48ccbcc, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.605595 1646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.605835 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.605938 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.606001 1646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: configured and running, proceeding with master startup.
W20240501 07:58:43.608106 1662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:43.608178 1662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:43.627270 1514 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } attempt: 1
I20240501 07:58:43.631665 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: Bootstrap starting.
I20240501 07:58:43.632436 1514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.633308 1514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: No bootstrap required, opened a new log
I20240501 07:58:43.633620 1514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.633708 1514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.633741 1514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f018f2612d4349b8041f62903ccf41, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.633837 1514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.634081 1664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.634166 1580 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } attempt: 1
I20240501 07:58:43.634199 1664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.634202 1514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:43.635924 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 07:58:43.636463 1675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:43.636512 1675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:43.638196 1580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: Bootstrap starting.
I20240501 07:58:43.638939 1580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.639775 1580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: No bootstrap required, opened a new log
I20240501 07:58:43.640118 1580 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.640198 1580 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.640229 1580 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd156b4cbcda4d2fb95b85fe24485255, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.640344 1580 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.640601 1677 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.640681 1677 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.640736 1580 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:43.642071 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 07:58:43.642489 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:43.642498 1688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:43.642568 1688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 07:58:43.644048 1690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.644560 1689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.644997 1692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.645462 396 server_base.cc:1034] running on GCE node
I20240501 07:58:43.645601 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.645634 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.645648 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323645647 us; error 0 us; skew 500 ppm
I20240501 07:58:43.646529 396 webserver.cc:460] Webserver started at http://127.0.99.1:39229/ using document root <none> and password file <none>
I20240501 07:58:43.646680 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.646726 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.646778 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.647112 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "e9b943ed43834b19b96afec3a903a09e"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.648387 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:43.649204 1697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.649399 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.649456 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "e9b943ed43834b19b96afec3a903a09e"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.649520 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:43.665074 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.665390 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.665616 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:43.666036 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:43.666072 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.666102 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:43.666131 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.670346 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:44743
I20240501 07:58:43.670389 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:44743 every 8 connection(s)
I20240501 07:58:43.671258 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:43.675740 1768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.676051 1769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.676375 1761 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39011
I20240501 07:58:43.676396 1762 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37659
I20240501 07:58:43.676466 1761 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.676502 1762 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.676689 1761 heartbeater.cc:511] Master 127.0.99.62:39011 requested a full tablet report, sending...
I20240501 07:58:43.676712 1763 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37973
I20240501 07:58:43.676730 1762 heartbeater.cc:511] Master 127.0.99.60:37659 requested a full tablet report, sending...
I20240501 07:58:43.676800 1763 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.677091 1763 heartbeater.cc:511] Master 127.0.99.61:37973 requested a full tablet report, sending...
W20240501 07:58:43.677140 1771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.677261 396 server_base.cc:1034] running on GCE node
I20240501 07:58:43.677426 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.677464 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.677482 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323677482 us; error 0 us; skew 500 ppm
I20240501 07:58:43.677552 1478 ts_manager.cc:194] Registered new tserver with Master: e9b943ed43834b19b96afec3a903a09e (127.0.99.1:44743)
I20240501 07:58:43.677889 1544 ts_manager.cc:194] Registered new tserver with Master: e9b943ed43834b19b96afec3a903a09e (127.0.99.1:44743)
I20240501 07:58:43.678156 1610 ts_manager.cc:194] Registered new tserver with Master: e9b943ed43834b19b96afec3a903a09e (127.0.99.1:44743)
I20240501 07:58:43.678817 396 webserver.cc:460] Webserver started at http://127.0.99.2:39555/ using document root <none> and password file <none>
I20240501 07:58:43.678953 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.679003 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.679067 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.679414 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "066157414f914ff58e727bbb4a60527e"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.680711 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:43.681535 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.681751 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.681811 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "066157414f914ff58e727bbb4a60527e"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.681867 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:43.693688 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.694057 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.694320 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:43.694818 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:43.694856 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.694896 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:43.694924 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.699069 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:41623
I20240501 07:58:43.699123 1839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:41623 every 8 connection(s)
I20240501 07:58:43.699960 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:43.702268 1847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:43.703188 1848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.703756 396 server_base.cc:1034] running on GCE node
I20240501 07:58:43.703976 1841 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37659
I20240501 07:58:43.704058 1841 heartbeater.cc:464] Registering TS with master...
W20240501 07:58:43.704217 1850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:43.704267 1841 heartbeater.cc:511] Master 127.0.99.60:37659 requested a full tablet report, sending...
I20240501 07:58:43.704430 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:43.704493 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:43.704505 1842 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37973
I20240501 07:58:43.704545 1840 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39011
I20240501 07:58:43.704589 1842 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.704622 1840 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.704521 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550323704521 us; error 0 us; skew 500 ppm
I20240501 07:58:43.704777 1842 heartbeater.cc:511] Master 127.0.99.61:37973 requested a full tablet report, sending...
I20240501 07:58:43.704821 1840 heartbeater.cc:511] Master 127.0.99.62:39011 requested a full tablet report, sending...
I20240501 07:58:43.704977 1610 ts_manager.cc:194] Registered new tserver with Master: 066157414f914ff58e727bbb4a60527e (127.0.99.2:41623)
I20240501 07:58:43.705339 1478 ts_manager.cc:194] Registered new tserver with Master: 066157414f914ff58e727bbb4a60527e (127.0.99.2:41623)
I20240501 07:58:43.705339 1544 ts_manager.cc:194] Registered new tserver with Master: 066157414f914ff58e727bbb4a60527e (127.0.99.2:41623)
I20240501 07:58:43.705818 396 webserver.cc:460] Webserver started at http://127.0.99.3:39145/ using document root <none> and password file <none>
I20240501 07:58:43.705955 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:43.705996 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:43.706048 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:43.706358 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "1031e80017374b51b9440316413487cd"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.707778 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:43.708524 1855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.708727 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.708786 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "1031e80017374b51b9440316413487cd"
format_stamp: "Formatted at 2024-05-01 07:58:43 on dist-test-slave-rp4l"
I20240501 07:58:43.708842 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:43.715365 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:43.715695 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:43.715950 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:43.716343 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:43.716387 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.716424 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:43.716451 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:43.720535 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:35553
I20240501 07:58:43.720585 1918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:35553 every 8 connection(s)
I20240501 07:58:43.725231 1920 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37659
I20240501 07:58:43.725324 1920 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.725370 1921 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37973
I20240501 07:58:43.725544 1921 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.725549 1919 heartbeater.cc:347] Connected to a master server at 127.0.99.62:39011
I20240501 07:58:43.725705 1921 heartbeater.cc:511] Master 127.0.99.61:37973 requested a full tablet report, sending...
I20240501 07:58:43.725697 1920 heartbeater.cc:511] Master 127.0.99.60:37659 requested a full tablet report, sending...
I20240501 07:58:43.725694 1919 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:43.725988 1919 heartbeater.cc:511] Master 127.0.99.62:39011 requested a full tablet report, sending...
I20240501 07:58:43.726364 1610 ts_manager.cc:194] Registered new tserver with Master: 1031e80017374b51b9440316413487cd (127.0.99.3:35553)
I20240501 07:58:43.726368 1544 ts_manager.cc:194] Registered new tserver with Master: 1031e80017374b51b9440316413487cd (127.0.99.3:35553)
I20240501 07:58:43.726496 1478 ts_manager.cc:194] Registered new tserver with Master: 1031e80017374b51b9440316413487cd (127.0.99.3:35553)
I20240501 07:58:43.726604 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005663767s
I20240501 07:58:43.738329 1664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.738448 1664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.738878 1664 leader_election.cc:288] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd156b4cbcda4d2fb95b85fe24485255 (127.0.99.61:37973), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
I20240501 07:58:43.739249 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f018f2612d4349b8041f62903ccf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd156b4cbcda4d2fb95b85fe24485255" is_pre_election: true
I20240501 07:58:43.739388 1554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f018f2612d4349b8041f62903ccf41 in term 0.
I20240501 07:58:43.739630 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f018f2612d4349b8041f62903ccf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc" is_pre_election: true
I20240501 07:58:43.739689 1465 leader_election.cc:302] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f018f2612d4349b8041f62903ccf41, fd156b4cbcda4d2fb95b85fe24485255; no voters:
I20240501 07:58:43.739756 1620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f018f2612d4349b8041f62903ccf41 in term 0.
I20240501 07:58:43.739881 1664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.739949 1664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.739988 1664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.740782 1664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.741104 1664 leader_election.cc:288] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [CANDIDATE]: Term 1 election: Requested vote from peers fd156b4cbcda4d2fb95b85fe24485255 (127.0.99.61:37973), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
I20240501 07:58:43.741371 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f018f2612d4349b8041f62903ccf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc"
I20240501 07:58:43.741487 1620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.742209 1620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f018f2612d4349b8041f62903ccf41 in term 1.
I20240501 07:58:43.742374 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f018f2612d4349b8041f62903ccf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd156b4cbcda4d2fb95b85fe24485255"
I20240501 07:58:43.742441 1464 leader_election.cc:302] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f018f2612d4349b8041f62903ccf41, 95d2a53816544ba7880a474cc48ccbcc; no voters:
I20240501 07:58:43.742458 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.742678 1664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.742880 1664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 1 LEADER]: Becoming Leader. State: Replica: 44f018f2612d4349b8041f62903ccf41, State: Running, Role: LEADER
I20240501 07:58:43.743018 1664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:43.743288 1554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f018f2612d4349b8041f62903ccf41 in term 1.
I20240501 07:58:43.743607 1927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f018f2612d4349b8041f62903ccf41. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.743693 1927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:43.743952 1929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:43.744606 1929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:43.746063 1554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Refusing update from remote peer 44f018f2612d4349b8041f62903ccf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.746073 1620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Refusing update from remote peer 44f018f2612d4349b8041f62903ccf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.746387 1927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.746538 1664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.747661 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f018f2612d4349b8041f62903ccf41. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.747666 1677 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f018f2612d4349b8041f62903ccf41. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.747793 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.747820 1677 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.749087 1664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.749119 1927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.749178 1664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:43.749207 1927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:43.749238 1929 catalog_manager.cc:1347] Generated new cluster ID: 9d3e3121f3c147999c5c756ee9e73330
I20240501 07:58:43.749293 1929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:43.749778 1677 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.749866 1677 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.750367 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f018f2612d4349b8041f62903ccf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:43.750465 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:43.767405 1929 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:43.768036 1929 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:43.776945 1929 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: Generated new TSK 0
I20240501 07:58:43.777125 1929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:44.608903 1662 catalog_manager.cc:1259] Loaded cluster ID: 9d3e3121f3c147999c5c756ee9e73330
I20240501 07:58:44.608989 1662 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: loading cluster ID for follower catalog manager: success
I20240501 07:58:44.610158 1662 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: acquiring CA information for follower catalog manager: success
I20240501 07:58:44.610704 1662 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 07:58:44.643245 1688 catalog_manager.cc:1259] Loaded cluster ID: 9d3e3121f3c147999c5c756ee9e73330
I20240501 07:58:44.643328 1688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: loading cluster ID for follower catalog manager: success
I20240501 07:58:44.644448 1688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: acquiring CA information for follower catalog manager: success
I20240501 07:58:44.644981 1688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 07:58:44.679984 1478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39694
I20240501 07:58:44.706971 1478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39702
I20240501 07:58:44.728253 1478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39710
I20240501 07:58:45.636025 1674 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:45.680763 1761 heartbeater.cc:503] Master 127.0.99.62:39011 was elected leader, sending a full tablet report...
I20240501 07:58:45.707732 1840 heartbeater.cc:503] Master 127.0.99.62:39011 was elected leader, sending a full tablet report...
I20240501 07:58:45.729089 1919 heartbeater.cc:503] Master 127.0.99.62:39011 was elected leader, sending a full tablet report...
I20240501 07:58:45.877979 396 test_util.cc:268] Using random seed: -618685225
I20240501 07:58:45.886318 1478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39716:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 07:58:45.898603 1804 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4326282a87451f85c8a998bb87da24 (DEFAULT_TABLE table=test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 07:58:45.898972 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4326282a87451f85c8a998bb87da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.899127 1883 tablet_service.cc:1467] Processing CreateTablet for tablet 770353b2ccb848b0ad336bba0c87fa5c (DEFAULT_TABLE table=test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 07:58:45.899346 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770353b2ccb848b0ad336bba0c87fa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.900671 1966 tablet_bootstrap.cc:492] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: Bootstrap starting.
I20240501 07:58:45.901028 1725 tablet_service.cc:1467] Processing CreateTablet for tablet d3363a6f684c4159b698a7ef1106a9cd (DEFAULT_TABLE table=test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 07:58:45.901201 1967 tablet_bootstrap.cc:492] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: Bootstrap starting.
I20240501 07:58:45.901278 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3363a6f684c4159b698a7ef1106a9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.902065 1967 tablet_bootstrap.cc:654] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.903077 1967 tablet_bootstrap.cc:492] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: No bootstrap required, opened a new log
I20240501 07:58:45.903158 1967 ts_tablet_manager.cc:1367] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:45.903380 1966 tablet_bootstrap.cc:654] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.903581 1967 raft_consensus.cc:348] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031e80017374b51b9440316413487cd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35553 } }
I20240501 07:58:45.903688 1967 raft_consensus.cc:374] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.903752 1967 raft_consensus.cc:729] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031e80017374b51b9440316413487cd, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.903870 1967 consensus_queue.cc:260] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031e80017374b51b9440316413487cd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35553 } }
I20240501 07:58:45.903976 1969 tablet_bootstrap.cc:492] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: Bootstrap starting.
I20240501 07:58:45.903996 1967 raft_consensus.cc:388] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:45.904352 1966 tablet_bootstrap.cc:492] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: No bootstrap required, opened a new log
I20240501 07:58:45.904438 1967 raft_consensus.cc:482] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:45.904489 1967 raft_consensus.cc:3037] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.904513 1966 ts_tablet_manager.cc:1367] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:45.904892 1966 raft_consensus.cc:348] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066157414f914ff58e727bbb4a60527e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41623 } }
I20240501 07:58:45.904969 1966 raft_consensus.cc:374] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.904997 1966 raft_consensus.cc:729] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066157414f914ff58e727bbb4a60527e, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.905081 1967 raft_consensus.cc:504] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031e80017374b51b9440316413487cd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35553 } }
I20240501 07:58:45.905141 1969 tablet_bootstrap.cc:654] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.905107 1966 consensus_queue.cc:260] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066157414f914ff58e727bbb4a60527e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41623 } }
I20240501 07:58:45.905182 1966 raft_consensus.cc:388] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:45.905187 1967 leader_election.cc:302] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1031e80017374b51b9440316413487cd; no voters:
I20240501 07:58:45.905242 1966 raft_consensus.cc:482] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:45.905285 1966 raft_consensus.cc:3037] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.905370 1967 leader_election.cc:288] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:45.905488 1971 raft_consensus.cc:2781] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.905630 1971 raft_consensus.cc:686] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 1 LEADER]: Becoming Leader. State: Replica: 1031e80017374b51b9440316413487cd, State: Running, Role: LEADER
I20240501 07:58:45.905684 1967 ts_tablet_manager.cc:1398] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:45.905740 1971 consensus_queue.cc:237] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031e80017374b51b9440316413487cd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35553 } }
I20240501 07:58:45.906111 1969 tablet_bootstrap.cc:492] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: No bootstrap required, opened a new log
I20240501 07:58:45.906082 1966 raft_consensus.cc:504] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066157414f914ff58e727bbb4a60527e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41623 } }
I20240501 07:58:45.906172 1969 ts_tablet_manager.cc:1367] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:45.906184 1966 leader_election.cc:302] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 066157414f914ff58e727bbb4a60527e; no voters:
I20240501 07:58:45.906345 1966 leader_election.cc:288] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:45.906594 1966 ts_tablet_manager.cc:1398] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:45.907344 1478 catalog_manager.cc:5484] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1031e80017374b51b9440316413487cd (127.0.99.3). New cstate: current_term: 1 leader_uuid: "1031e80017374b51b9440316413487cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1031e80017374b51b9440316413487cd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35553 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:45.908264 1969 raft_consensus.cc:348] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b943ed43834b19b96afec3a903a09e" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44743 } }
I20240501 07:58:45.908412 1969 raft_consensus.cc:374] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.908490 1969 raft_consensus.cc:729] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b943ed43834b19b96afec3a903a09e, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.908681 1969 consensus_queue.cc:260] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b943ed43834b19b96afec3a903a09e" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44743 } }
I20240501 07:58:45.908828 1969 raft_consensus.cc:388] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:45.908908 1969 raft_consensus.cc:482] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:45.908991 1969 raft_consensus.cc:3037] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.912410 1969 raft_consensus.cc:504] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b943ed43834b19b96afec3a903a09e" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44743 } }
I20240501 07:58:45.912516 1969 leader_election.cc:302] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b943ed43834b19b96afec3a903a09e; no voters:
I20240501 07:58:45.912683 1969 leader_election.cc:288] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:45.912770 1978 raft_consensus.cc:2781] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.913074 1978 raft_consensus.cc:686] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 1 LEADER]: Becoming Leader. State: Replica: e9b943ed43834b19b96afec3a903a09e, State: Running, Role: LEADER
I20240501 07:58:45.913092 1969 ts_tablet_manager.cc:1398] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:45.913286 1976 raft_consensus.cc:2781] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.913452 1978 consensus_queue.cc:237] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b943ed43834b19b96afec3a903a09e" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44743 } }
I20240501 07:58:45.913558 1976 raft_consensus.cc:686] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 1 LEADER]: Becoming Leader. State: Replica: 066157414f914ff58e727bbb4a60527e, State: Running, Role: LEADER
I20240501 07:58:45.913677 1976 consensus_queue.cc:237] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066157414f914ff58e727bbb4a60527e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41623 } }
I20240501 07:58:45.914659 1478 catalog_manager.cc:5484] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e reported cstate change: term changed from 0 to 1, leader changed from <none> to e9b943ed43834b19b96afec3a903a09e (127.0.99.1). New cstate: current_term: 1 leader_uuid: "e9b943ed43834b19b96afec3a903a09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b943ed43834b19b96afec3a903a09e" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44743 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:45.914865 1477 catalog_manager.cc:5484] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e reported cstate change: term changed from 0 to 1, leader changed from <none> to 066157414f914ff58e727bbb4a60527e (127.0.99.2). New cstate: current_term: 1 leader_uuid: "066157414f914ff58e727bbb4a60527e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066157414f914ff58e727bbb4a60527e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41623 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:45.926334 396 master.cc:537] Master@127.0.99.62:39011 shutting down...
I20240501 07:58:45.928990 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:45.929183 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:45.929235 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 44f018f2612d4349b8041f62903ccf41: stopping tablet replica
I20240501 07:58:45.942243 396 master.cc:552] Master@127.0.99.62:39011 shutdown complete.
W20240501 07:58:46.913426 1919 heartbeater.cc:644] Failed to heartbeat to 127.0.99.62:39011 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:47.421195 1990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44f018f2612d4349b8041f62903ccf41)
I20240501 07:58:47.421263 1991 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44f018f2612d4349b8041f62903ccf41)
I20240501 07:58:47.421339 1990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:47.421357 1991 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:47.421775 1991 leader_election.cc:288] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
I20240501 07:58:47.421901 1990 leader_election.cc:288] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), fd156b4cbcda4d2fb95b85fe24485255 (127.0.99.61:37973)
I20240501 07:58:47.422127 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d2a53816544ba7880a474cc48ccbcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fd156b4cbcda4d2fb95b85fe24485255" is_pre_election: true
I20240501 07:58:47.422235 1554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d2a53816544ba7880a474cc48ccbcc in term 1.
I20240501 07:58:47.422322 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd156b4cbcda4d2fb95b85fe24485255" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc" is_pre_election: true
I20240501 07:58:47.422431 1620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd156b4cbcda4d2fb95b85fe24485255 in term 1.
I20240501 07:58:47.422643 1596 leader_election.cc:302] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d2a53816544ba7880a474cc48ccbcc, fd156b4cbcda4d2fb95b85fe24485255; no voters:
I20240501 07:58:47.422838 1990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:58:47.422878 1530 leader_election.cc:302] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d2a53816544ba7880a474cc48ccbcc, fd156b4cbcda4d2fb95b85fe24485255; no voters:
I20240501 07:58:47.422909 1990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44f018f2612d4349b8041f62903ccf41)
I20240501 07:58:47.422979 1990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:47.423022 1530 leader_election.cc:334] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:47.423094 1991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:58:47.423134 1991 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44f018f2612d4349b8041f62903ccf41)
I20240501 07:58:47.423170 1991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:47.423270 1595 leader_election.cc:334] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:47.423897 1990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:47.423951 1991 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:47.424211 1991 leader_election.cc:288] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 election: Requested vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
I20240501 07:58:47.424278 1990 leader_election.cc:288] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 election: Requested vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), fd156b4cbcda4d2fb95b85fe24485255 (127.0.99.61:37973)
I20240501 07:58:47.424598 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d2a53816544ba7880a474cc48ccbcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fd156b4cbcda4d2fb95b85fe24485255"
I20240501 07:58:47.424638 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd156b4cbcda4d2fb95b85fe24485255" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc"
I20240501 07:58:47.424711 1554 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 95d2a53816544ba7880a474cc48ccbcc in current term 2: Already voted for candidate fd156b4cbcda4d2fb95b85fe24485255 in this term.
I20240501 07:58:47.424747 1620 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd156b4cbcda4d2fb95b85fe24485255 in current term 2: Already voted for candidate 95d2a53816544ba7880a474cc48ccbcc in this term.
W20240501 07:58:47.425187 1595 leader_election.cc:334] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:47.425254 1595 leader_election.cc:302] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95d2a53816544ba7880a474cc48ccbcc; no voters: 44f018f2612d4349b8041f62903ccf41, fd156b4cbcda4d2fb95b85fe24485255
W20240501 07:58:47.425298 1530 leader_election.cc:334] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:47.425360 1990 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:58:47.425384 1530 leader_election.cc:302] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd156b4cbcda4d2fb95b85fe24485255; no voters: 44f018f2612d4349b8041f62903ccf41, 95d2a53816544ba7880a474cc48ccbcc
I20240501 07:58:47.425544 1991 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:58:48.995759 2002 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:48.995898 2002 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:48.996306 2002 leader_election.cc:288] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
W20240501 07:58:48.996888 1530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 07:58:48.997028 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd156b4cbcda4d2fb95b85fe24485255" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc" is_pre_election: true
I20240501 07:58:48.997144 1620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd156b4cbcda4d2fb95b85fe24485255 in term 2.
I20240501 07:58:48.997509 1530 leader_election.cc:302] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d2a53816544ba7880a474cc48ccbcc, fd156b4cbcda4d2fb95b85fe24485255; no voters:
W20240501 07:58:48.997642 1530 leader_election.cc:334] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:48.997678 2002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 07:58:48.997741 2002 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:48.997784 2002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:58:48.998584 2002 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:48.998904 2002 leader_election.cc:288] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 election: Requested vote from peers 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011), 95d2a53816544ba7880a474cc48ccbcc (127.0.99.60:37659)
I20240501 07:58:48.999316 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd156b4cbcda4d2fb95b85fe24485255" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "95d2a53816544ba7880a474cc48ccbcc"
I20240501 07:58:48.999397 1620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:58:48.999634 1530 leader_election.cc:334] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111)
I20240501 07:58:49.000053 1620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd156b4cbcda4d2fb95b85fe24485255 in term 3.
I20240501 07:58:49.000276 1530 leader_election.cc:302] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95d2a53816544ba7880a474cc48ccbcc, fd156b4cbcda4d2fb95b85fe24485255; no voters: 44f018f2612d4349b8041f62903ccf41
I20240501 07:58:49.000416 2002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:58:49.000577 2002 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 3 LEADER]: Becoming Leader. State: Replica: fd156b4cbcda4d2fb95b85fe24485255, State: Running, Role: LEADER
I20240501 07:58:49.000705 2002 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } }
I20240501 07:58:49.001232 2005 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd156b4cbcda4d2fb95b85fe24485255. Latest consensus state: current_term: 3 leader_uuid: "fd156b4cbcda4d2fb95b85fe24485255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:49.001332 2005 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:49.001567 2007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:49.002575 2007 catalog_manager.cc:669] Loaded metadata for table test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0]
I20240501 07:58:49.003337 2007 tablet_loader.cc:96] loaded metadata for tablet 0c4326282a87451f85c8a998bb87da24 (table test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0])
I20240501 07:58:49.003523 2007 tablet_loader.cc:96] loaded metadata for tablet 770353b2ccb848b0ad336bba0c87fa5c (table test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0])
I20240501 07:58:49.003685 2007 tablet_loader.cc:96] loaded metadata for tablet d3363a6f684c4159b698a7ef1106a9cd (table test-workload [id=2d1ccd8512ee4b1ba06be6ac3abcdde0])
I20240501 07:58:49.003775 2007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:49.004148 2007 catalog_manager.cc:1259] Loaded cluster ID: 9d3e3121f3c147999c5c756ee9e73330
I20240501 07:58:49.004189 2007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:49.004688 2007 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:49.005052 2007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: Loaded TSK: 0
I20240501 07:58:49.005498 2007 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 07:58:49.423398 1530 consensus_peers.cc:485] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 -> Peer 44f018f2612d4349b8041f62903ccf41 (127.0.99.62:39011): Couldn't send request to peer 44f018f2612d4349b8041f62903ccf41. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.62:39011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:49.483392 1620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 3 FOLLOWER]: Refusing update from remote peer fd156b4cbcda4d2fb95b85fe24485255: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 07:58:49.483810 2005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:49.484743 2008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd156b4cbcda4d2fb95b85fe24485255. Latest consensus state: current_term: 3 leader_uuid: "fd156b4cbcda4d2fb95b85fe24485255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:49.484848 2008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:49.485550 2002 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fd156b4cbcda4d2fb95b85fe24485255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:49.485656 2002 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:49.485970 2008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fd156b4cbcda4d2fb95b85fe24485255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f018f2612d4349b8041f62903ccf41" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 39011 } } peers { permanent_uuid: "fd156b4cbcda4d2fb95b85fe24485255" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37973 } } peers { permanent_uuid: "95d2a53816544ba7880a474cc48ccbcc" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37659 } } }
I20240501 07:58:49.486063 2008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 07:58:49.642741 1687 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:49.642843 1687 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:50.079185 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 07:58:50.082409 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:50.082588 396 tablet_replica.cc:330] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e: stopping tablet replica
I20240501 07:58:50.082729 396 raft_consensus.cc:2229] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:50.082859 396 raft_consensus.cc:2258] T d3363a6f684c4159b698a7ef1106a9cd P e9b943ed43834b19b96afec3a903a09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.095279 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:58:50.096961 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 07:58:50.100149 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:50.100308 396 tablet_replica.cc:330] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e: stopping tablet replica
I20240501 07:58:50.100404 396 raft_consensus.cc:2229] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:50.100497 396 raft_consensus.cc:2258] T 0c4326282a87451f85c8a998bb87da24 P 066157414f914ff58e727bbb4a60527e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.112460 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 07:58:50.114092 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 07:58:50.117133 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:58:50.117280 396 tablet_replica.cc:330] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd: stopping tablet replica
I20240501 07:58:50.117377 396 raft_consensus.cc:2229] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:50.117537 396 raft_consensus.cc:2258] T 770353b2ccb848b0ad336bba0c87fa5c P 1031e80017374b51b9440316413487cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.129549 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:58:50.131248 396 master.cc:537] Master@127.0.99.61:37973 shutting down...
I20240501 07:58:50.133911 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 3 LEADER]: Raft consensus shutting down.
I20240501 07:58:50.134150 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.134212 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd156b4cbcda4d2fb95b85fe24485255: stopping tablet replica
I20240501 07:58:50.146553 396 master.cc:552] Master@127.0.99.61:37973 shutdown complete.
I20240501 07:58:50.148887 396 master.cc:537] Master@127.0.99.60:37659 shutting down...
I20240501 07:58:50.151762 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.151885 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.151932 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 95d2a53816544ba7880a474cc48ccbcc: stopping tablet replica
I20240501 07:58:50.163874 396 master.cc:552] Master@127.0.99.60:37659 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6655 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 07:58:50.169870 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:36167
I20240501 07:58:50.170150 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.172250 2017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.172395 2020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.172911 2018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.173422 396 server_base.cc:1034] running on GCE node
I20240501 07:58:50.173537 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:50.173571 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:50.173590 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550330173590 us; error 0 us; skew 500 ppm
I20240501 07:58:50.179041 396 webserver.cc:460] Webserver started at http://127.0.99.62:33769/ using document root <none> and password file <none>
I20240501 07:58:50.179172 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.179219 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.179287 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.179627 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "5e5760f4da134a4b8a08cf98c8fabcb7"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.180871 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 07:58:50.181649 2025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.181890 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:50.181949 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "5e5760f4da134a4b8a08cf98c8fabcb7"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.182011 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.189352 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.189625 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.193157 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:36167
I20240501 07:58:50.195384 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:36167 every 8 connection(s)
I20240501 07:58:50.195767 2078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.197332 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7: Bootstrap starting.
I20240501 07:58:50.198010 2078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.198869 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7: No bootstrap required, opened a new log
I20240501 07:58:50.199195 2078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER }
I20240501 07:58:50.199275 2078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.199303 2078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5760f4da134a4b8a08cf98c8fabcb7, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.199414 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER }
I20240501 07:58:50.199477 2078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:50.199506 2078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:50.199549 2078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.200119 2078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER }
I20240501 07:58:50.200228 2078 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5760f4da134a4b8a08cf98c8fabcb7; no voters:
I20240501 07:58:50.200374 2078 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:50.200454 2081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.200620 2081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5760f4da134a4b8a08cf98c8fabcb7, State: Running, Role: LEADER
I20240501 07:58:50.200759 2078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:50.200749 2081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER }
I20240501 07:58:50.201138 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER } }
I20240501 07:58:50.201174 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5760f4da134a4b8a08cf98c8fabcb7. Latest consensus state: current_term: 1 leader_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5760f4da134a4b8a08cf98c8fabcb7" member_type: VOTER } }
I20240501 07:58:50.201285 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:50.201303 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:50.201709 2086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:50.202386 2086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:50.202598 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:58:50.203941 2086 catalog_manager.cc:1347] Generated new cluster ID: b40fd0766a0148d0bcd5d0de057284c9
I20240501 07:58:50.203999 2086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:50.227480 2086 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:50.228001 2086 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:50.237053 2086 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7: Generated new TSK 0
I20240501 07:58:50.237203 2086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:50.267040 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.268652 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.269179 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.269702 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.270143 396 server_base.cc:1034] running on GCE node
I20240501 07:58:50.270280 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:50.270316 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:50.270335 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550330270335 us; error 0 us; skew 500 ppm
I20240501 07:58:50.271191 396 webserver.cc:460] Webserver started at http://127.0.99.1:45199/ using document root <none> and password file <none>
I20240501 07:58:50.271324 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.271371 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.271446 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.271817 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "c20094b167004cc3a392e4bdc9c9b5f9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.273072 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:50.273797 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.273995 396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 07:58:50.274056 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "c20094b167004cc3a392e4bdc9c9b5f9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.274116 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.282653 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.282948 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.283185 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.283596 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.283632 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.283668 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.283695 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.287281 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:42633
I20240501 07:58:50.287329 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:42633 every 8 connection(s)
I20240501 07:58:50.287966 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.289284 2175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.289748 2176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.290223 2178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.290693 396 server_base.cc:1034] running on GCE node
I20240501 07:58:50.290838 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:50.290885 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:50.290920 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550330290919 us; error 0 us; skew 500 ppm
I20240501 07:58:50.291692 396 webserver.cc:460] Webserver started at http://127.0.99.2:42805/ using document root <none> and password file <none>
I20240501 07:58:50.291837 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.291889 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.291952 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.292137 2171 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36167
I20240501 07:58:50.292222 2171 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.292315 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "31a0c5590567493aa03db54fcb319aa9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.292433 2171 heartbeater.cc:511] Master 127.0.99.62:36167 requested a full tablet report, sending...
I20240501 07:58:50.292977 2042 ts_manager.cc:194] Registered new tserver with Master: c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1:42633)
I20240501 07:58:50.293614 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:50.293648 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42716
I20240501 07:58:50.294463 2183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.294679 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.294735 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "31a0c5590567493aa03db54fcb319aa9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.294798 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.305931 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.306223 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.306473 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.306911 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.306946 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.306977 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.307003 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.310529 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:33345
I20240501 07:58:50.310585 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:33345 every 8 connection(s)
I20240501 07:58:50.311273 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.312717 2250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.313103 2251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.313539 396 server_base.cc:1034] running on GCE node
W20240501 07:58:50.313926 2253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.314100 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:58:50.314148 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:58:50.314167 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550330314167 us; error 0 us; skew 500 ppm
I20240501 07:58:50.315060 396 webserver.cc:460] Webserver started at http://127.0.99.3:43745/ using document root <none> and password file <none>
I20240501 07:58:50.315073 2247 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36167
I20240501 07:58:50.315171 2247 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.315215 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.315264 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.315327 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.315383 2247 heartbeater.cc:511] Master 127.0.99.62:36167 requested a full tablet report, sending...
I20240501 07:58:50.315737 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "a5c1404c3c2e466692ca76271075ebf8"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.315925 2042 ts_manager.cc:194] Registered new tserver with Master: 31a0c5590567493aa03db54fcb319aa9 (127.0.99.2:33345)
I20240501 07:58:50.316411 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42724
I20240501 07:58:50.317081 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:50.317828 2258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.318058 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.318113 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "a5c1404c3c2e466692ca76271075ebf8"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-rp4l"
I20240501 07:58:50.318163 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.325675 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.325929 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.326133 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.326488 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.326520 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.326545 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.326565 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.330451 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:36741
I20240501 07:58:50.330497 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:36741 every 8 connection(s)
I20240501 07:58:50.334201 2322 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36167
I20240501 07:58:50.334288 2322 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.334465 2322 heartbeater.cc:511] Master 127.0.99.62:36167 requested a full tablet report, sending...
I20240501 07:58:50.335005 2042 ts_manager.cc:194] Registered new tserver with Master: a5c1404c3c2e466692ca76271075ebf8 (127.0.99.3:36741)
I20240501 07:58:50.335249 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004424996s
I20240501 07:58:50.335501 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42728
I20240501 07:58:51.294481 2171 heartbeater.cc:503] Master 127.0.99.62:36167 was elected leader, sending a full tablet report...
I20240501 07:58:51.317219 2247 heartbeater.cc:503] Master 127.0.99.62:36167 was elected leader, sending a full tablet report...
I20240501 07:58:51.336212 2322 heartbeater.cc:503] Master 127.0.99.62:36167 was elected leader, sending a full tablet report...
I20240501 07:58:51.360769 396 test_util.cc:268] Using random seed: -613202436
I20240501 07:58:51.367821 2042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42736:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 07:58:51.368237 2042 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 07:58:51.377260 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 04e93b62b07e4f78a4afddc468b359b1 (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 07:58:51.377614 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e93b62b07e4f78a4afddc468b359b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.378791 2211 tablet_service.cc:1467] Processing CreateTablet for tablet 89741b95da824155a8aa11d3a934735b (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 07:58:51.378952 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 89741b95da824155a8aa11d3a934735b (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 07:58:51.378959 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 04e93b62b07e4f78a4afddc468b359b1 (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 07:58:51.379050 2211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89741b95da824155a8aa11d3a934735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.379330 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e93b62b07e4f78a4afddc468b359b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.379981 2342 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Bootstrap starting.
I20240501 07:58:51.380139 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 04e93b62b07e4f78a4afddc468b359b1 (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 07:58:51.380375 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e93b62b07e4f78a4afddc468b359b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.381000 2342 tablet_bootstrap.cc:654] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.381274 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89741b95da824155a8aa11d3a934735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.382292 2342 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: No bootstrap required, opened a new log
I20240501 07:58:51.377261 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 89741b95da824155a8aa11d3a934735b (DEFAULT_TABLE table=test-workload [id=8624b29146814497b2e3300eee9acbb0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 07:58:51.382438 2342 ts_tablet_manager.cc:1367] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:51.382588 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89741b95da824155a8aa11d3a934735b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.383075 2343 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: Bootstrap starting.
I20240501 07:58:51.383476 2342 raft_consensus.cc:348] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.383605 2342 raft_consensus.cc:374] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.383677 2342 raft_consensus.cc:729] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20094b167004cc3a392e4bdc9c9b5f9, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.383844 2342 consensus_queue.cc:260] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.383952 2343 tablet_bootstrap.cc:654] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.384267 2342 ts_tablet_manager.cc:1398] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:51.384477 2342 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: Bootstrap starting.
I20240501 07:58:51.384567 2344 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: Bootstrap starting.
I20240501 07:58:51.385259 2344 tablet_bootstrap.cc:654] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.385422 2342 tablet_bootstrap.cc:654] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.386389 2343 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: No bootstrap required, opened a new log
I20240501 07:58:51.386466 2343 ts_tablet_manager.cc:1367] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:51.386482 2344 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: No bootstrap required, opened a new log
I20240501 07:58:51.386512 2342 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: No bootstrap required, opened a new log
I20240501 07:58:51.386560 2344 ts_tablet_manager.cc:1367] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:51.386569 2342 ts_tablet_manager.cc:1367] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:51.386865 2343 raft_consensus.cc:348] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.386965 2343 raft_consensus.cc:374] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.387003 2343 raft_consensus.cc:729] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c1404c3c2e466692ca76271075ebf8, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.387099 2343 consensus_queue.cc:260] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.387185 2342 raft_consensus.cc:348] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.387253 2344 raft_consensus.cc:348] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.387307 2342 raft_consensus.cc:374] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.387326 2344 raft_consensus.cc:374] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.387341 2342 raft_consensus.cc:729] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20094b167004cc3a392e4bdc9c9b5f9, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.387357 2344 raft_consensus.cc:729] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a0c5590567493aa03db54fcb319aa9, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.387398 2343 ts_tablet_manager.cc:1398] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.387455 2342 consensus_queue.cc:260] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.387480 2344 consensus_queue.cc:260] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.387533 2343 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: Bootstrap starting.
I20240501 07:58:51.387709 2342 ts_tablet_manager.cc:1398] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.388069 2344 ts_tablet_manager.cc:1398] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.388321 2343 tablet_bootstrap.cc:654] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.389438 2343 tablet_bootstrap.cc:492] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: No bootstrap required, opened a new log
I20240501 07:58:51.389561 2343 ts_tablet_manager.cc:1367] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:51.389696 2344 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: Bootstrap starting.
I20240501 07:58:51.389971 2343 raft_consensus.cc:348] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.390118 2343 raft_consensus.cc:374] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.390188 2343 raft_consensus.cc:729] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c1404c3c2e466692ca76271075ebf8, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.390347 2346 raft_consensus.cc:482] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.390360 2343 consensus_queue.cc:260] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.390483 2346 raft_consensus.cc:504] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.390532 2344 tablet_bootstrap.cc:654] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.391274 2343 ts_tablet_manager.cc:1398] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:51.391748 2346 leader_election.cc:288] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5c1404c3c2e466692ca76271075ebf8 (127.0.99.3:36741), 31a0c5590567493aa03db54fcb319aa9 (127.0.99.2:33345)
I20240501 07:58:51.392532 2344 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: No bootstrap required, opened a new log
I20240501 07:58:51.392619 2344 ts_tablet_manager.cc:1367] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:51.392995 2344 raft_consensus.cc:348] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.393096 2344 raft_consensus.cc:374] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.393127 2344 raft_consensus.cc:729] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a0c5590567493aa03db54fcb319aa9, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.393230 2344 consensus_queue.cc:260] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.393463 2344 ts_tablet_manager.cc:1398] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:51.395467 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89741b95da824155a8aa11d3a934735b" candidate_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1404c3c2e466692ca76271075ebf8" is_pre_election: true
I20240501 07:58:51.395593 2296 raft_consensus.cc:2454] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20094b167004cc3a392e4bdc9c9b5f9 in term 0.
I20240501 07:58:51.395937 2109 leader_election.cc:302] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c1404c3c2e466692ca76271075ebf8, c20094b167004cc3a392e4bdc9c9b5f9; no voters:
I20240501 07:58:51.396124 2346 raft_consensus.cc:2781] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.396198 2346 raft_consensus.cc:482] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.396237 2346 raft_consensus.cc:3037] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.396373 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89741b95da824155a8aa11d3a934735b" candidate_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a0c5590567493aa03db54fcb319aa9" is_pre_election: true
I20240501 07:58:51.396457 2221 raft_consensus.cc:2454] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20094b167004cc3a392e4bdc9c9b5f9 in term 0.
I20240501 07:58:51.396994 2346 raft_consensus.cc:504] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.397272 2346 leader_election.cc:288] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [CANDIDATE]: Term 1 election: Requested vote from peers a5c1404c3c2e466692ca76271075ebf8 (127.0.99.3:36741), 31a0c5590567493aa03db54fcb319aa9 (127.0.99.2:33345)
I20240501 07:58:51.397473 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89741b95da824155a8aa11d3a934735b" candidate_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1404c3c2e466692ca76271075ebf8"
I20240501 07:58:51.397563 2296 raft_consensus.cc:3037] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.397572 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89741b95da824155a8aa11d3a934735b" candidate_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a0c5590567493aa03db54fcb319aa9"
I20240501 07:58:51.397684 2221 raft_consensus.cc:3037] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.398265 2296 raft_consensus.cc:2454] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20094b167004cc3a392e4bdc9c9b5f9 in term 1.
I20240501 07:58:51.398362 2221 raft_consensus.cc:2454] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20094b167004cc3a392e4bdc9c9b5f9 in term 1.
I20240501 07:58:51.398476 2109 leader_election.cc:302] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c1404c3c2e466692ca76271075ebf8, c20094b167004cc3a392e4bdc9c9b5f9; no voters:
I20240501 07:58:51.398649 2346 raft_consensus.cc:2781] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.398813 2346 raft_consensus.cc:686] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 LEADER]: Becoming Leader. State: Replica: c20094b167004cc3a392e4bdc9c9b5f9, State: Running, Role: LEADER
I20240501 07:58:51.398929 2346 consensus_queue.cc:237] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } }
I20240501 07:58:51.400117 2041 catalog_manager.cc:5484] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:51.435663 2351 raft_consensus.cc:482] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.435798 2351 raft_consensus.cc:504] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.436262 2351 leader_election.cc:288] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1:42633), a5c1404c3c2e466692ca76271075ebf8 (127.0.99.3:36741)
I20240501 07:58:51.439473 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e93b62b07e4f78a4afddc468b359b1" candidate_uuid: "31a0c5590567493aa03db54fcb319aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1404c3c2e466692ca76271075ebf8" is_pre_election: true
I20240501 07:58:51.439513 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e93b62b07e4f78a4afddc468b359b1" candidate_uuid: "31a0c5590567493aa03db54fcb319aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" is_pre_election: true
I20240501 07:58:51.439595 2296 raft_consensus.cc:2454] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a0c5590567493aa03db54fcb319aa9 in term 0.
I20240501 07:58:51.439612 2145 raft_consensus.cc:2454] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a0c5590567493aa03db54fcb319aa9 in term 0.
I20240501 07:58:51.439831 2185 leader_election.cc:302] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a0c5590567493aa03db54fcb319aa9, a5c1404c3c2e466692ca76271075ebf8; no voters:
I20240501 07:58:51.439980 2351 raft_consensus.cc:2781] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.440035 2351 raft_consensus.cc:482] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.440078 2351 raft_consensus.cc:3037] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.440819 2351 raft_consensus.cc:504] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.441097 2351 leader_election.cc:288] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [CANDIDATE]: Term 1 election: Requested vote from peers c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1:42633), a5c1404c3c2e466692ca76271075ebf8 (127.0.99.3:36741)
I20240501 07:58:51.441326 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e93b62b07e4f78a4afddc468b359b1" candidate_uuid: "31a0c5590567493aa03db54fcb319aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20094b167004cc3a392e4bdc9c9b5f9"
I20240501 07:58:51.441339 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e93b62b07e4f78a4afddc468b359b1" candidate_uuid: "31a0c5590567493aa03db54fcb319aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1404c3c2e466692ca76271075ebf8"
I20240501 07:58:51.441440 2145 raft_consensus.cc:3037] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.441440 2296 raft_consensus.cc:3037] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.442245 2145 raft_consensus.cc:2454] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a0c5590567493aa03db54fcb319aa9 in term 1.
I20240501 07:58:51.442245 2296 raft_consensus.cc:2454] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a0c5590567493aa03db54fcb319aa9 in term 1.
I20240501 07:58:51.442499 2185 leader_election.cc:302] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a0c5590567493aa03db54fcb319aa9, a5c1404c3c2e466692ca76271075ebf8; no voters:
I20240501 07:58:51.442670 2351 raft_consensus.cc:2781] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.442847 2351 raft_consensus.cc:686] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 31a0c5590567493aa03db54fcb319aa9, State: Running, Role: LEADER
I20240501 07:58:51.442965 2351 consensus_queue.cc:237] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } }
I20240501 07:58:51.444248 2041 catalog_manager.cc:5484] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a0c5590567493aa03db54fcb319aa9 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "31a0c5590567493aa03db54fcb319aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:51.777639 2346 consensus_queue.cc:1035] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.780362 2346 consensus_queue.cc:1035] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.835314 2351 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.837919 2360 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.202524 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:52.202917 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:52.202988 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:54.203250 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:54.203655 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:54.203727 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:56.204046 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:56.204456 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:56.204524 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:58:58.204818 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:58:58.205209 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:58:58.205279 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:00.205560 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:00.205933 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:00.205998 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:02.206274 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:02.206688 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:02.206755 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:02.504624 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.506918 2378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.507508 2380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.507575 2377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.508145 396 server_base.cc:1034] running on GCE node
I20240501 07:59:02.508301 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:02.508339 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:02.508354 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550342508354 us; error 0 us; skew 500 ppm
I20240501 07:59:02.513096 396 webserver.cc:460] Webserver started at http://127.0.99.4:36341/ using document root <none> and password file <none>
I20240501 07:59:02.513235 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.513283 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.513357 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.513720 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-3-root/instance:
uuid: "8dd0ca5e35814466b03b3432f0bfdc25"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-rp4l"
I20240501 07:59:02.515027 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:02.515847 2385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.516070 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.516139 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-3-root
uuid: "8dd0ca5e35814466b03b3432f0bfdc25"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-rp4l"
I20240501 07:59:02.516196 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.532557 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.532917 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.533174 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:02.533596 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:02.533633 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.533672 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:02.533699 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.537753 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:43125
I20240501 07:59:02.537811 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:43125 every 8 connection(s)
I20240501 07:59:02.541747 2449 heartbeater.cc:347] Connected to a master server at 127.0.99.62:36167
I20240501 07:59:02.541822 2449 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:02.542008 2449 heartbeater.cc:511] Master 127.0.99.62:36167 requested a full tablet report, sending...
I20240501 07:59:02.542519 2041 ts_manager.cc:194] Registered new tserver with Master: 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4:43125)
I20240501 07:59:02.543041 2041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41802
I20240501 07:59:03.216153 2221 consensus_queue.cc:237] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } }
I20240501 07:59:03.217470 2145 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.217587 2296 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.217965 2457 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.218158 2456 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.219702 2375 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } } }
I20240501 07:59:03.220045 2145 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } } }
I20240501 07:59:03.221527 2296 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } } }
W20240501 07:59:03.222024 2186 consensus_peers.cc:485] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 -> Peer 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4:43125): Couldn't send request to peer 8dd0ca5e35814466b03b3432f0bfdc25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04e93b62b07e4f78a4afddc468b359b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.222035 2041 catalog_manager.cc:5484] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 reported cstate change: config changed from index -1 to 2, NON_VOTER 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "31a0c5590567493aa03db54fcb319aa9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:03.543741 2449 heartbeater.cc:503] Master 127.0.99.62:36167 was elected leader, sending a full tablet report...
I20240501 07:59:03.679630 2467 ts_tablet_manager.cc:902] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Initiating tablet copy from peer 31a0c5590567493aa03db54fcb319aa9 (127.0.99.2:33345)
I20240501 07:59:03.680542 2467 tablet_copy_client.cc:308] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:33345
I20240501 07:59:03.683977 2231 tablet_copy_service.cc:140] P 31a0c5590567493aa03db54fcb319aa9: Received BeginTabletCopySession request for tablet 04e93b62b07e4f78a4afddc468b359b1 from peer 8dd0ca5e35814466b03b3432f0bfdc25 ({username='slave'} at 127.0.0.1:36448)
I20240501 07:59:03.684082 2231 tablet_copy_service.cc:161] P 31a0c5590567493aa03db54fcb319aa9: Beginning new tablet copy session on tablet 04e93b62b07e4f78a4afddc468b359b1 from peer 8dd0ca5e35814466b03b3432f0bfdc25 at {username='slave'} at 127.0.0.1:36448: session id = 8dd0ca5e35814466b03b3432f0bfdc25-04e93b62b07e4f78a4afddc468b359b1
I20240501 07:59:03.685124 2231 tablet_copy_source_session.cc:205] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:03.686311 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e93b62b07e4f78a4afddc468b359b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.688482 2467 tablet_copy_client.cc:783] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:03.688654 2467 tablet_copy_client.cc:647] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:03.689827 2467 tablet_copy_client.cc:523] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:03.691290 2467 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Bootstrap starting.
I20240501 07:59:03.695313 2467 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:03.695538 2467 tablet_bootstrap.cc:492] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Bootstrap complete.
I20240501 07:59:03.695675 2467 ts_tablet_manager.cc:1367] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 07:59:03.696095 2467 raft_consensus.cc:348] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } }
I20240501 07:59:03.696209 2467 raft_consensus.cc:729] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8dd0ca5e35814466b03b3432f0bfdc25, State: Initialized, Role: LEARNER
I20240501 07:59:03.696308 2467 consensus_queue.cc:260] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: true } }
I20240501 07:59:03.696662 2467 ts_tablet_manager.cc:1398] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:03.697098 2231 tablet_copy_service.cc:342] P 31a0c5590567493aa03db54fcb319aa9: Request end of tablet copy session 8dd0ca5e35814466b03b3432f0bfdc25-04e93b62b07e4f78a4afddc468b359b1 received from {username='slave'} at 127.0.0.1:36448
I20240501 07:59:03.697177 2231 tablet_copy_service.cc:433] P 31a0c5590567493aa03db54fcb319aa9: ending tablet copy session 8dd0ca5e35814466b03b3432f0bfdc25-04e93b62b07e4f78a4afddc468b359b1 on tablet 04e93b62b07e4f78a4afddc468b359b1 with peer 8dd0ca5e35814466b03b3432f0bfdc25
I20240501 07:59:04.206948 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:04.207325 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:04.207394 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:04.301038 2423 raft_consensus.cc:1203] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:04.871624 2478 raft_consensus.cc:1050] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: attempting to promote NON_VOTER 8dd0ca5e35814466b03b3432f0bfdc25 to VOTER
I20240501 07:59:04.872144 2478 consensus_queue.cc:237] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } }
I20240501 07:59:04.873342 2423 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 LEARNER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:04.873450 2296 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:04.873447 2145 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:04.873733 2481 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:04.873932 2478 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:04.874378 2479 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:04.875627 2478 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.876443 2145 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.877480 2423 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.877633 2296 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.878247 2042 catalog_manager.cc:5484] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 reported cstate change: config changed from index 2 to 3, 8dd0ca5e35814466b03b3432f0bfdc25 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31a0c5590567493aa03db54fcb319aa9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c20094b167004cc3a392e4bdc9c9b5f9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.882494 2220 consensus_queue.cc:237] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } }
I20240501 07:59:04.883409 2423 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:04.883486 2296 raft_consensus.cc:1261] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Refusing update from remote peer 31a0c5590567493aa03db54fcb319aa9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:04.883718 2482 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:04.883911 2481 consensus_queue.cc:1035] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:04.885128 2481 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.885597 2296 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.885594 2423 raft_consensus.cc:2932] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } } }
I20240501 07:59:04.887238 2026 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04e93b62b07e4f78a4afddc468b359b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:04.887544 2040 catalog_manager.cc:5484] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 reported cstate change: config changed from index 3 to 4, VOTER c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "31a0c5590567493aa03db54fcb319aa9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31a0c5590567493aa03db54fcb319aa9" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5c1404c3c2e466692ca76271075ebf8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dd0ca5e35814466b03b3432f0bfdc25" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:04.889588 2135 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e93b62b07e4f78a4afddc468b359b1 with delete_type TABLET_DATA_TOMBSTONED (TS c20094b167004cc3a392e4bdc9c9b5f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51712
I20240501 07:59:04.890064 2492 tablet_replica.cc:330] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: stopping tablet replica
I20240501 07:59:04.890185 2492 raft_consensus.cc:2229] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:04.890309 2492 raft_consensus.cc:2258] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:04.890794 2492 ts_tablet_manager.cc:1875] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:04.892410 2492 ts_tablet_manager.cc:1888] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:04.892478 2492 log.cc:1196] T 04e93b62b07e4f78a4afddc468b359b1 P c20094b167004cc3a392e4bdc9c9b5f9: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714550313795535-396-0/minicluster-data/ts-0-root/wals/04e93b62b07e4f78a4afddc468b359b1
I20240501 07:59:04.892957 2029 catalog_manager.cc:4830] TS c20094b167004cc3a392e4bdc9c9b5f9 (127.0.99.1:42633): tablet 04e93b62b07e4f78a4afddc468b359b1 (table test-workload [id=8624b29146814497b2e3300eee9acbb0]) successfully deleted
I20240501 07:59:06.207705 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:06.208112 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:06.208186 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:08.208483 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:08.208890 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:08.208971 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:10.209343 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:10.209751 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:10.209817 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:12.210142 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:12.210587 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:12.210681 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:14.210986 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:14.211396 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:14.211478 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:16.211812 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:16.212225 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:16.212292 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:16.749042 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 07:59:16.752629 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:16.752823 396 tablet_replica.cc:330] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9: stopping tablet replica
I20240501 07:59:16.752938 396 raft_consensus.cc:2229] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:16.753141 396 raft_consensus.cc:2258] T 89741b95da824155a8aa11d3a934735b P c20094b167004cc3a392e4bdc9c9b5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.765813 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:59:16.767812 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 07:59:16.771232 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:16.771404 396 tablet_replica.cc:330] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9: stopping tablet replica
I20240501 07:59:16.771526 396 raft_consensus.cc:2229] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:16.771721 396 raft_consensus.cc:2258] T 04e93b62b07e4f78a4afddc468b359b1 P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.772125 396 tablet_replica.cc:330] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9: stopping tablet replica
I20240501 07:59:16.772200 396 raft_consensus.cc:2229] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.772289 396 raft_consensus.cc:2258] T 89741b95da824155a8aa11d3a934735b P 31a0c5590567493aa03db54fcb319aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.784822 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 07:59:16.787029 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 07:59:16.790900 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:16.791083 396 tablet_replica.cc:330] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8: stopping tablet replica
I20240501 07:59:16.791186 396 raft_consensus.cc:2229] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.791299 396 raft_consensus.cc:2258] T 89741b95da824155a8aa11d3a934735b P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.791703 396 tablet_replica.cc:330] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8: stopping tablet replica
I20240501 07:59:16.791779 396 raft_consensus.cc:2229] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.791896 396 raft_consensus.cc:2258] T 04e93b62b07e4f78a4afddc468b359b1 P a5c1404c3c2e466692ca76271075ebf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.804114 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:59:16.806155 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240501 07:59:16.809700 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:16.809860 396 tablet_replica.cc:330] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25: stopping tablet replica
I20240501 07:59:16.809962 396 raft_consensus.cc:2229] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.810073 396 raft_consensus.cc:2258] T 04e93b62b07e4f78a4afddc468b359b1 P 8dd0ca5e35814466b03b3432f0bfdc25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.822165 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240501 07:59:16.824088 396 master.cc:537] Master@127.0.99.62:36167 shutting down...
I20240501 07:59:16.827304 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:16.827458 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.827517 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e5760f4da134a4b8a08cf98c8fabcb7: stopping tablet replica
I20240501 07:59:16.840898 396 master.cc:552] Master@127.0.99.62:36167 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26677 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 07:59:16.847482 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:32781
I20240501 07:59:16.847798 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:16.849915 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:16.850061 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:16.850720 396 server_base.cc:1034] running on GCE node
W20240501 07:59:16.851171 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:16.851341 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:16.851379 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:16.851393 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550356851393 us; error 0 us; skew 500 ppm
I20240501 07:59:16.852198 396 webserver.cc:460] Webserver started at http://127.0.99.62:40985/ using document root <none> and password file <none>
I20240501 07:59:16.852317 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:16.852353 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:16.852411 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:16.852752 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "9e2e7b794d304c78a7c7065f83163840"
format_stamp: "Formatted at 2024-05-01 07:59:16 on dist-test-slave-rp4l"
I20240501 07:59:16.853986 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.854766 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:16.854992 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.855067 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "9e2e7b794d304c78a7c7065f83163840"
format_stamp: "Formatted at 2024-05-01 07:59:16 on dist-test-slave-rp4l"
I20240501 07:59:16.855129 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:16.874982 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:16.875327 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:16.878930 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:32781
I20240501 07:59:16.880961 2557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:32781 every 8 connection(s)
I20240501 07:59:16.881368 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.883087 2558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840: Bootstrap starting.
I20240501 07:59:16.883775 2558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.884662 2558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840: No bootstrap required, opened a new log
I20240501 07:59:16.884984 2558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER }
I20240501 07:59:16.885068 2558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.885099 2558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2e7b794d304c78a7c7065f83163840, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.885214 2558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER }
I20240501 07:59:16.885284 2558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:16.885311 2558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:16.885344 2558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.885905 2558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER }
I20240501 07:59:16.886008 2558 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2e7b794d304c78a7c7065f83163840; no voters:
I20240501 07:59:16.886145 2558 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:16.886241 2561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.886435 2561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2e7b794d304c78a7c7065f83163840, State: Running, Role: LEADER
I20240501 07:59:16.886535 2558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:16.886579 2561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER }
I20240501 07:59:16.887009 2562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2e7b794d304c78a7c7065f83163840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER } }
I20240501 07:59:16.887037 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2e7b794d304c78a7c7065f83163840. Latest consensus state: current_term: 1 leader_uuid: "9e2e7b794d304c78a7c7065f83163840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2e7b794d304c78a7c7065f83163840" member_type: VOTER } }
I20240501 07:59:16.887173 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:16.887408 2567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:16.887634 2562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:16.888257 2567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:16.888393 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:59:16.889958 2567 catalog_manager.cc:1347] Generated new cluster ID: 36d29f42539b42848de6d4b7c5e35d20
I20240501 07:59:16.890008 2567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:16.897181 2567 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:16.897682 2567 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:16.916316 2567 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840: Generated new TSK 0
I20240501 07:59:16.916488 2567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:16.920233 396 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 5.2639e-05s
I20240501 07:59:28.997117 396 master.cc:537] Master@127.0.99.62:32781 shutting down...
I20240501 07:59:28.999958 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:29.000121 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:29.000174 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9e2e7b794d304c78a7c7065f83163840: stopping tablet replica
I20240501 07:59:29.011976 396 master.cc:552] Master@127.0.99.62:32781 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12168 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 07:59:29.015702 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43217
I20240501 07:59:29.015995 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:29.018087 2583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.018236 2581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.018800 2580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:29.019289 396 server_base.cc:1034] running on GCE node
I20240501 07:59:29.019412 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:29.019444 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:29.019462 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550369019462 us; error 0 us; skew 500 ppm
I20240501 07:59:29.020318 396 webserver.cc:460] Webserver started at http://127.0.99.62:36789/ using document root <none> and password file <none>
I20240501 07:59:29.020447 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:29.020488 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:29.020555 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:29.020872 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "5e596c4db93d48578689d5f592c01267"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.022107 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.022923 2588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.023164 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.023221 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "5e596c4db93d48578689d5f592c01267"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.023273 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:29.034114 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:29.034365 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:29.037801 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43217
I20240501 07:59:29.038339 2640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43217 every 8 connection(s)
I20240501 07:59:29.038754 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:29.040345 2641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267: Bootstrap starting.
I20240501 07:59:29.041019 2641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:29.041927 2641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267: No bootstrap required, opened a new log
I20240501 07:59:29.042270 2641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER }
I20240501 07:59:29.042352 2641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:29.042383 2641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e596c4db93d48578689d5f592c01267, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.042505 2641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER }
I20240501 07:59:29.042565 2641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:29.042596 2641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:29.042657 2641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:29.043246 2641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER }
I20240501 07:59:29.043360 2641 leader_election.cc:302] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e596c4db93d48578689d5f592c01267; no voters:
I20240501 07:59:29.043530 2641 leader_election.cc:288] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:29.043627 2644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:29.043802 2644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 1 LEADER]: Becoming Leader. State: Replica: 5e596c4db93d48578689d5f592c01267, State: Running, Role: LEADER
I20240501 07:59:29.043908 2641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:29.043942 2644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER }
I20240501 07:59:29.044322 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e596c4db93d48578689d5f592c01267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER } }
I20240501 07:59:29.044405 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:29.044361 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e596c4db93d48578689d5f592c01267. Latest consensus state: current_term: 1 leader_uuid: "5e596c4db93d48578689d5f592c01267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e596c4db93d48578689d5f592c01267" member_type: VOTER } }
I20240501 07:59:29.044446 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:29.044618 2649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:29.045414 2649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:29.045670 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:59:29.047088 2649 catalog_manager.cc:1347] Generated new cluster ID: a66a4b0674c74e75bd277d1f9c6b132f
I20240501 07:59:29.047133 2649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:29.056509 2649 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:29.057009 2649 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:29.063494 2649 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267: Generated new TSK 0
I20240501 07:59:29.063627 2649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:29.077797 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:29.079877 2665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.080408 2663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.080536 2662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:29.080976 396 server_base.cc:1034] running on GCE node
I20240501 07:59:29.081099 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:29.081135 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:29.081147 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550369081147 us; error 0 us; skew 500 ppm
I20240501 07:59:29.081962 396 webserver.cc:460] Webserver started at http://127.0.99.1:43101/ using document root <none> and password file <none>
I20240501 07:59:29.082091 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:29.082130 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:29.082199 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:29.082532 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "032b97be26124567929a7a2377d0c4b8"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.083853 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:29.084623 2670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.084837 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:29.084893 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "032b97be26124567929a7a2377d0c4b8"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.084955 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:29.105618 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:29.105944 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:29.106201 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:29.106662 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:29.106704 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.106737 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:29.106765 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.110422 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:33137
I20240501 07:59:29.110459 2733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:33137 every 8 connection(s)
I20240501 07:59:29.111193 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:29.112690 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.113139 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.113631 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:29.114078 396 server_base.cc:1034] running on GCE node
I20240501 07:59:29.114204 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:29.114245 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:29.114259 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550369114259 us; error 0 us; skew 500 ppm
I20240501 07:59:29.115130 396 webserver.cc:460] Webserver started at http://127.0.99.2:42069/ using document root <none> and password file <none>
I20240501 07:59:29.115280 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:29.115334 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:29.115370 2734 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43217
I20240501 07:59:29.115403 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:29.115446 2734 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:29.115604 2734 heartbeater.cc:511] Master 127.0.99.62:43217 requested a full tablet report, sending...
I20240501 07:59:29.115706 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "be05f113b0c04cbbb7e93d662db40b65"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.116166 2605 ts_manager.cc:194] Registered new tserver with Master: 032b97be26124567929a7a2377d0c4b8 (127.0.99.1:33137)
I20240501 07:59:29.117100 2605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57946
I20240501 07:59:29.117218 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:29.118041 2746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.118275 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.118335 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "be05f113b0c04cbbb7e93d662db40b65"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.118398 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:29.134367 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:29.134687 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:29.134935 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:29.135329 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:29.135365 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.135394 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:29.135421 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.139125 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:36005
I20240501 07:59:29.139165 2809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:36005 every 8 connection(s)
I20240501 07:59:29.139825 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:29.141634 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.142099 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:29.142457 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:29.142954 396 server_base.cc:1034] running on GCE node
I20240501 07:59:29.143103 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:29.143152 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:29.143172 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550369143172 us; error 0 us; skew 500 ppm
I20240501 07:59:29.143580 2810 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43217
I20240501 07:59:29.143668 2810 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:29.143868 2810 heartbeater.cc:511] Master 127.0.99.62:43217 requested a full tablet report, sending...
I20240501 07:59:29.144125 396 webserver.cc:460] Webserver started at http://127.0.99.3:39073/ using document root <none> and password file <none>
I20240501 07:59:29.144266 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:29.144313 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:29.144392 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:29.144479 2605 ts_manager.cc:194] Registered new tserver with Master: be05f113b0c04cbbb7e93d662db40b65 (127.0.99.2:36005)
I20240501 07:59:29.144824 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "87403aeef2534fafb6ca2ebd47e1af65"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.145097 2605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57950
I20240501 07:59:29.146251 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:29.147084 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.147316 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:29.147373 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "87403aeef2534fafb6ca2ebd47e1af65"
format_stamp: "Formatted at 2024-05-01 07:59:29 on dist-test-slave-rp4l"
I20240501 07:59:29.147423 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:29.167636 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:29.168006 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:29.168267 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:29.168720 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:29.168759 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.168799 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:29.168828 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:29.173081 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:42627
I20240501 07:59:29.173137 2884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:42627 every 8 connection(s)
I20240501 07:59:29.176959 2885 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43217
I20240501 07:59:29.177039 2885 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:29.177229 2885 heartbeater.cc:511] Master 127.0.99.62:43217 requested a full tablet report, sending...
I20240501 07:59:29.177762 2605 ts_manager.cc:194] Registered new tserver with Master: 87403aeef2534fafb6ca2ebd47e1af65 (127.0.99.3:42627)
I20240501 07:59:29.177956 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004495774s
I20240501 07:59:29.178414 2605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57960
I20240501 07:59:30.117945 2734 heartbeater.cc:503] Master 127.0.99.62:43217 was elected leader, sending a full tablet report...
I20240501 07:59:30.145888 2810 heartbeater.cc:503] Master 127.0.99.62:43217 was elected leader, sending a full tablet report...
I20240501 07:59:30.179209 2885 heartbeater.cc:503] Master 127.0.99.62:43217 was elected leader, sending a full tablet report...
I20240501 07:59:31.045724 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:33.046066 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:35.046434 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:37.046780 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:39.047123 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:41.047493 2659 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:41.254451 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 07:59:41.258077 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:41.269909 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:59:41.271756 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 07:59:41.274875 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:41.286615 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 07:59:41.288359 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 07:59:41.291143 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:41.302878 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:59:41.304560 396 master.cc:537] Master@127.0.99.62:43217 shutting down...
I20240501 07:59:41.307484 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:41.307652 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:41.307704 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e596c4db93d48578689d5f592c01267: stopping tablet replica
I20240501 07:59:41.319550 396 master.cc:552] Master@127.0.99.62:43217 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12308 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 07:59:41.324260 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:45035
I20240501 07:59:41.324540 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.326726 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.326834 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.327411 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.328022 396 server_base.cc:1034] running on GCE node
I20240501 07:59:41.328186 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:41.328223 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:41.328238 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550381328237 us; error 0 us; skew 500 ppm
I20240501 07:59:41.331894 396 webserver.cc:460] Webserver started at http://127.0.99.62:37281/ using document root <none> and password file <none>
I20240501 07:59:41.332033 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.332077 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.332151 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.332523 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "f874d18afc63445ab534b6802fee40c3"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.333930 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:41.334898 2897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.335116 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:41.335197 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "f874d18afc63445ab534b6802fee40c3"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.335265 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.339668 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.339967 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.343632 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:45035
I20240501 07:59:41.344736 2949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:45035 every 8 connection(s)
I20240501 07:59:41.345147 2950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:41.346854 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3: Bootstrap starting.
I20240501 07:59:41.347553 2950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:41.348407 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3: No bootstrap required, opened a new log
I20240501 07:59:41.348742 2950 raft_consensus.cc:348] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER }
I20240501 07:59:41.348826 2950 raft_consensus.cc:374] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:41.348855 2950 raft_consensus.cc:729] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f874d18afc63445ab534b6802fee40c3, State: Initialized, Role: FOLLOWER
I20240501 07:59:41.348984 2950 consensus_queue.cc:260] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER }
I20240501 07:59:41.349062 2950 raft_consensus.cc:388] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:41.349100 2950 raft_consensus.cc:482] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:41.349144 2950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:41.349748 2950 raft_consensus.cc:504] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER }
I20240501 07:59:41.349864 2950 leader_election.cc:302] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f874d18afc63445ab534b6802fee40c3; no voters:
I20240501 07:59:41.350029 2950 leader_election.cc:288] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:41.350113 2953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:41.350322 2953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 1 LEADER]: Becoming Leader. State: Replica: f874d18afc63445ab534b6802fee40c3, State: Running, Role: LEADER
I20240501 07:59:41.350425 2950 sys_catalog.cc:564] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:41.350451 2953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER }
I20240501 07:59:41.350878 2954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f874d18afc63445ab534b6802fee40c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER } }
I20240501 07:59:41.350977 2954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:41.350893 2955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f874d18afc63445ab534b6802fee40c3. Latest consensus state: current_term: 1 leader_uuid: "f874d18afc63445ab534b6802fee40c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f874d18afc63445ab534b6802fee40c3" member_type: VOTER } }
I20240501 07:59:41.351032 2955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:41.351203 2958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:41.351984 2958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:41.352136 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:59:41.353639 2958 catalog_manager.cc:1347] Generated new cluster ID: 11895d174ca948008776b3f4531a2c4e
I20240501 07:59:41.353691 2958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:41.360863 2958 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:41.361394 2958 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:41.371685 2958 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3: Generated new TSK 0
I20240501 07:59:41.371862 2958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:41.384413 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.386652 2971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.387153 2974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.387264 2972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.387776 396 server_base.cc:1034] running on GCE node
I20240501 07:59:41.387907 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:41.387943 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:41.387956 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550381387956 us; error 0 us; skew 500 ppm
I20240501 07:59:41.388797 396 webserver.cc:460] Webserver started at http://127.0.99.1:33045/ using document root <none> and password file <none>
I20240501 07:59:41.388918 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.388957 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.389015 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.389335 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "93ada6555b4849e888e8472f9a32d7c3"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.390691 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:41.391503 2979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.391726 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:41.391785 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "93ada6555b4849e888e8472f9a32d7c3"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.391847 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.408380 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.408743 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.409008 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.409437 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.409476 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.409518 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.409544 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.413313 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:37555
I20240501 07:59:41.413343 3042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:37555 every 8 connection(s)
I20240501 07:59:41.414037 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.415570 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.416038 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.416502 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.416922 396 server_base.cc:1034] running on GCE node
I20240501 07:59:41.417070 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:41.417119 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:41.417151 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550381417150 us; error 0 us; skew 500 ppm
I20240501 07:59:41.418180 396 webserver.cc:460] Webserver started at http://127.0.99.2:33293/ using document root <none> and password file <none>
I20240501 07:59:41.418335 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.418401 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.418470 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.418704 3043 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45035
I20240501 07:59:41.418789 3043 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.418887 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "8e353072483d4c4d8059ce05b42d7815"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.419018 3043 heartbeater.cc:511] Master 127.0.99.62:45035 requested a full tablet report, sending...
I20240501 07:59:41.419607 2914 ts_manager.cc:194] Registered new tserver with Master: 93ada6555b4849e888e8472f9a32d7c3 (127.0.99.1:37555)
I20240501 07:59:41.420298 2914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44970
I20240501 07:59:41.420326 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:41.421281 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.421489 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:41.421550 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "8e353072483d4c4d8059ce05b42d7815"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.421613 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.438998 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.439363 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.439633 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.440071 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.440112 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.440155 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.440184 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.444439 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:40071
I20240501 07:59:41.444474 3118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:40071 every 8 connection(s)
I20240501 07:59:41.445212 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.447113 3123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.447175 3122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.447600 3125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.448091 396 server_base.cc:1034] running on GCE node
I20240501 07:59:41.448236 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:41.448271 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:41.448290 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550381448290 us; error 0 us; skew 500 ppm
I20240501 07:59:41.449172 3119 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45035
I20240501 07:59:41.449236 396 webserver.cc:460] Webserver started at http://127.0.99.3:40251/ using document root <none> and password file <none>
I20240501 07:59:41.449270 3119 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.449378 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.449432 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.449486 3119 heartbeater.cc:511] Master 127.0.99.62:45035 requested a full tablet report, sending...
I20240501 07:59:41.449501 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.449977 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "f3c555459c134724932b55c500b81ebd"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.450093 2914 ts_manager.cc:194] Registered new tserver with Master: 8e353072483d4c4d8059ce05b42d7815 (127.0.99.2:40071)
I20240501 07:59:41.450690 2914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44980
I20240501 07:59:41.451629 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:41.452441 3130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.452670 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:41.452735 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "f3c555459c134724932b55c500b81ebd"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.452801 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.462160 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.462532 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.462846 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.463287 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.463325 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.463366 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.463393 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.467664 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:45401
I20240501 07:59:41.467710 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:45401 every 8 connection(s)
I20240501 07:59:41.468417 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.469905 3197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.470412 3198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.470896 3200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.472292 396 server_base.cc:1034] running on GCE node
I20240501 07:59:41.472465 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 07:59:41.472496 3194 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45035
W20240501 07:59:41.472514 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:41.472609 3194 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.472617 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550381472617 us; error 0 us; skew 500 ppm
I20240501 07:59:41.472838 3194 heartbeater.cc:511] Master 127.0.99.62:45035 requested a full tablet report, sending...
I20240501 07:59:41.473486 2914 ts_manager.cc:194] Registered new tserver with Master: f3c555459c134724932b55c500b81ebd (127.0.99.3:45401)
I20240501 07:59:41.473613 396 webserver.cc:460] Webserver started at http://127.0.99.4:45663/ using document root <none> and password file <none>
I20240501 07:59:41.473749 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.473801 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.473867 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.474139 2914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44982
I20240501 07:59:41.474309 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-3-root/instance:
uuid: "68c078011c984a55a32e710085b4850b"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.475725 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:41.476511 3205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.476716 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:41.476778 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-3-root
uuid: "68c078011c984a55a32e710085b4850b"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-rp4l"
I20240501 07:59:41.476837 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714550313795535-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.503366 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.503767 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.504048 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:41.504478 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:41.504516 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.504555 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:41.504582 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.509115 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:41593
I20240501 07:59:41.509164 3268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:41593 every 8 connection(s)
I20240501 07:59:41.513036 3269 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45035
I20240501 07:59:41.513125 3269 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:41.513327 3269 heartbeater.cc:511] Master 127.0.99.62:45035 requested a full tablet report, sending...
I20240501 07:59:41.513860 2914 ts_manager.cc:194] Registered new tserver with Master: 68c078011c984a55a32e710085b4850b (127.0.99.4:41593)
I20240501 07:59:41.513967 396 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.004483909s
I20240501 07:59:41.514497 2914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44998
I20240501 07:59:42.421175 3043 heartbeater.cc:503] Master 127.0.99.62:45035 was elected leader, sending a full tablet report...
I20240501 07:59:42.451596 3119 heartbeater.cc:503] Master 127.0.99.62:45035 was elected leader, sending a full tablet report...
I20240501 07:59:42.475040 3194 heartbeater.cc:503] Master 127.0.99.62:45035 was elected leader, sending a full tablet report...
I20240501 07:59:42.515328 3269 heartbeater.cc:503] Master 127.0.99.62:45035 was elected leader, sending a full tablet report...
I20240501 07:59:42.544337 396 test_util.cc:268] Using random seed: -562018867
I20240501 07:59:42.551237 2914 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45018:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 07:59:42.560985 3158 tablet_service.cc:1467] Processing CreateTablet for tablet d909e8634319447185c4dd9a63151b0b (DEFAULT_TABLE table=test-workload [id=7a9aeb8838f94033b6f1482ba9f81c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:42.561213 3233 tablet_service.cc:1467] Processing CreateTablet for tablet d909e8634319447185c4dd9a63151b0b (DEFAULT_TABLE table=test-workload [id=7a9aeb8838f94033b6f1482ba9f81c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:42.561313 3158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909e8634319447185c4dd9a63151b0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:42.561486 3233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909e8634319447185c4dd9a63151b0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:42.561594 3007 tablet_service.cc:1467] Processing CreateTablet for tablet d909e8634319447185c4dd9a63151b0b (DEFAULT_TABLE table=test-workload [id=7a9aeb8838f94033b6f1482ba9f81c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:42.561803 3007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909e8634319447185c4dd9a63151b0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:42.563658 3290 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: Bootstrap starting.
I20240501 07:59:42.563926 3291 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: Bootstrap starting.
I20240501 07:59:42.563926 3292 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: Bootstrap starting.
I20240501 07:59:42.564499 3290 tablet_bootstrap.cc:654] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:42.564895 3291 tablet_bootstrap.cc:654] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:42.564895 3292 tablet_bootstrap.cc:654] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:42.566025 3291 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: No bootstrap required, opened a new log
I20240501 07:59:42.566077 3292 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: No bootstrap required, opened a new log
I20240501 07:59:42.566079 3290 tablet_bootstrap.cc:492] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: No bootstrap required, opened a new log
I20240501 07:59:42.566157 3291 ts_tablet_manager.cc:1367] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:42.566196 3292 ts_tablet_manager.cc:1367] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:42.566196 3290 ts_tablet_manager.cc:1367] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:42.566613 3291 raft_consensus.cc:348] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.566686 3292 raft_consensus.cc:348] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.566744 3291 raft_consensus.cc:374] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:42.566792 3292 raft_consensus.cc:374] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:42.566818 3291 raft_consensus.cc:729] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ada6555b4849e888e8472f9a32d7c3, State: Initialized, Role: FOLLOWER
I20240501 07:59:42.566831 3292 raft_consensus.cc:729] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c555459c134724932b55c500b81ebd, State: Initialized, Role: FOLLOWER
I20240501 07:59:42.566967 3291 consensus_queue.cc:260] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.566951 3292 consensus_queue.cc:260] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.566954 3290 raft_consensus.cc:348] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.567194 3290 raft_consensus.cc:374] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:42.567231 3290 raft_consensus.cc:729] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c078011c984a55a32e710085b4850b, State: Initialized, Role: FOLLOWER
I20240501 07:59:42.567348 3290 consensus_queue.cc:260] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.567366 3291 ts_tablet_manager.cc:1398] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:42.567364 3292 ts_tablet_manager.cc:1398] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:42.567677 3290 ts_tablet_manager.cc:1398] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:59:42.664952 3044 tablet.cc:2369] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:42.734495 3297 raft_consensus.cc:482] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:42.734683 3297 raft_consensus.cc:504] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.735194 3297 leader_election.cc:288] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68c078011c984a55a32e710085b4850b (127.0.99.4:41593), 93ada6555b4849e888e8472f9a32d7c3 (127.0.99.1:37555)
I20240501 07:59:42.738951 3243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909e8634319447185c4dd9a63151b0b" candidate_uuid: "f3c555459c134724932b55c500b81ebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c078011c984a55a32e710085b4850b" is_pre_election: true
I20240501 07:59:42.738953 3017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909e8634319447185c4dd9a63151b0b" candidate_uuid: "f3c555459c134724932b55c500b81ebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ada6555b4849e888e8472f9a32d7c3" is_pre_election: true
I20240501 07:59:42.739109 3243 raft_consensus.cc:2454] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c555459c134724932b55c500b81ebd in term 0.
I20240501 07:59:42.739138 3017 raft_consensus.cc:2454] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c555459c134724932b55c500b81ebd in term 0.
I20240501 07:59:42.739413 3134 leader_election.cc:302] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93ada6555b4849e888e8472f9a32d7c3, f3c555459c134724932b55c500b81ebd; no voters:
I20240501 07:59:42.739573 3297 raft_consensus.cc:2781] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:42.739626 3297 raft_consensus.cc:482] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:42.739671 3297 raft_consensus.cc:3037] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:42.740499 3297 raft_consensus.cc:504] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.740808 3297 leader_election.cc:288] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [CANDIDATE]: Term 1 election: Requested vote from peers 68c078011c984a55a32e710085b4850b (127.0.99.4:41593), 93ada6555b4849e888e8472f9a32d7c3 (127.0.99.1:37555)
I20240501 07:59:42.741160 3243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909e8634319447185c4dd9a63151b0b" candidate_uuid: "f3c555459c134724932b55c500b81ebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c078011c984a55a32e710085b4850b"
I20240501 07:59:42.741160 3017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909e8634319447185c4dd9a63151b0b" candidate_uuid: "f3c555459c134724932b55c500b81ebd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ada6555b4849e888e8472f9a32d7c3"
I20240501 07:59:42.741259 3243 raft_consensus.cc:3037] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:42.741274 3017 raft_consensus.cc:3037] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:42.742079 3017 raft_consensus.cc:2454] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c555459c134724932b55c500b81ebd in term 1.
I20240501 07:59:42.742084 3243 raft_consensus.cc:2454] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c555459c134724932b55c500b81ebd in term 1.
I20240501 07:59:42.742342 3134 leader_election.cc:302] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c078011c984a55a32e710085b4850b, f3c555459c134724932b55c500b81ebd; no voters:
I20240501 07:59:42.742481 3297 raft_consensus.cc:2781] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:42.742686 3297 raft_consensus.cc:686] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 1 LEADER]: Becoming Leader. State: Replica: f3c555459c134724932b55c500b81ebd, State: Running, Role: LEADER
I20240501 07:59:42.742821 3297 consensus_queue.cc:237] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } }
I20240501 07:59:42.744267 2913 catalog_manager.cc:5484] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c555459c134724932b55c500b81ebd (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f3c555459c134724932b55c500b81ebd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c555459c134724932b55c500b81ebd" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:43.332569 3303 consensus_queue.cc:1035] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ada6555b4849e888e8472f9a32d7c3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:43.335234 3303 consensus_queue.cc:1035] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [LEADER]: Connected to new peer: Peer: permanent_uuid: "68c078011c984a55a32e710085b4850b" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:43.352294 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:43.352587 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:43.352655 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:45.352954 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:45.353262 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:45.353344 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:47.353688 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:47.354012 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:47.354082 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:49.354352 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:49.354707 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:49.354779 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:51.355073 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:51.355391 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:51.355469 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:53.355787 2968 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:53.356091 2968 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:53.356158 2968 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:53.845764 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 07:59:53.849076 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:53.849262 396 tablet_replica.cc:330] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3: stopping tablet replica
I20240501 07:59:53.849376 396 raft_consensus.cc:2229] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:53.849503 396 raft_consensus.cc:2258] T d909e8634319447185c4dd9a63151b0b P 93ada6555b4849e888e8472f9a32d7c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:53.862855 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 07:59:53.864701 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 07:59:53.868116 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:53.879814 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 07:59:53.881546 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 07:59:53.885179 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:53.885344 396 tablet_replica.cc:330] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd: stopping tablet replica
I20240501 07:59:53.885443 396 raft_consensus.cc:2229] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:53.885617 396 raft_consensus.cc:2258] T d909e8634319447185c4dd9a63151b0b P f3c555459c134724932b55c500b81ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:53.898090 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 07:59:53.900007 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240501 07:59:53.903518 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 07:59:53.903694 396 tablet_replica.cc:330] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b: stopping tablet replica
I20240501 07:59:53.903810 396 raft_consensus.cc:2229] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:53.903921 396 raft_consensus.cc:2258] T d909e8634319447185c4dd9a63151b0b P 68c078011c984a55a32e710085b4850b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:53.916177 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240501 07:59:53.918051 396 master.cc:537] Master@127.0.99.62:45035 shutting down...
I20240501 07:59:53.921198 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:53.921351 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:53.921402 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P f874d18afc63445ab534b6802fee40c3: stopping tablet replica
I20240501 07:59:53.934546 396 master.cc:552] Master@127.0.99.62:45035 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12617 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 07:59:53.941501 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:33059
I20240501 07:59:53.941793 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:53.943794 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.944368 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.944521 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.945027 396 server_base.cc:1034] running on GCE node
I20240501 07:59:53.945158 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:53.945194 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:53.945206 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550393945206 us; error 0 us; skew 500 ppm
I20240501 07:59:53.949102 396 webserver.cc:460] Webserver started at http://127.0.99.62:41261/ using document root <none> and password file <none>
I20240501 07:59:53.949242 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:53.949285 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:53.949357 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:53.949720 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "d783c0bf3fb0443bbd741ed1bc4154eb"
format_stamp: "Formatted at 2024-05-01 07:59:53 on dist-test-slave-rp4l"
I20240501 07:59:53.951100 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:53.951944 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.952159 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:53.952220 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "d783c0bf3fb0443bbd741ed1bc4154eb"
format_stamp: "Formatted at 2024-05-01 07:59:53 on dist-test-slave-rp4l"
I20240501 07:59:53.952284 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:53.959967 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:53.960310 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:53.964035 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:33059
I20240501 07:59:53.965662 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:33059 every 8 connection(s)
I20240501 07:59:53.966068 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:53.967782 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb: Bootstrap starting.
I20240501 07:59:53.968470 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:53.969319 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb: No bootstrap required, opened a new log
I20240501 07:59:53.969638 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER }
I20240501 07:59:53.969719 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:53.969748 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d783c0bf3fb0443bbd741ed1bc4154eb, State: Initialized, Role: FOLLOWER
I20240501 07:59:53.969869 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER }
I20240501 07:59:53.969942 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:53.969970 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:53.970003 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:53.970566 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER }
I20240501 07:59:53.970705 3375 leader_election.cc:302] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d783c0bf3fb0443bbd741ed1bc4154eb; no voters:
I20240501 07:59:53.970850 3375 leader_election.cc:288] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:53.970957 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:53.971148 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 1 LEADER]: Becoming Leader. State: Replica: d783c0bf3fb0443bbd741ed1bc4154eb, State: Running, Role: LEADER
I20240501 07:59:53.971261 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:53.971271 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER }
I20240501 07:59:53.971704 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER } }
I20240501 07:59:53.971735 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d783c0bf3fb0443bbd741ed1bc4154eb. Latest consensus state: current_term: 1 leader_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d783c0bf3fb0443bbd741ed1bc4154eb" member_type: VOTER } }
I20240501 07:59:53.971789 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:53.971813 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:53.972002 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:53.972757 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:53.972960 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 07:59:53.974360 3383 catalog_manager.cc:1347] Generated new cluster ID: e477ab84f07b4faf8a0e3bc4d26f79f8
I20240501 07:59:53.974404 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:53.993713 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:53.994259 3383 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:54.000877 3383 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb: Generated new TSK 0
I20240501 07:59:54.001041 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:54.005085 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:54.007154 3397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.007287 3396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.007764 3399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:54.008322 396 server_base.cc:1034] running on GCE node
I20240501 07:59:54.008471 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:54.008507 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:54.008525 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550394008525 us; error 0 us; skew 500 ppm
I20240501 07:59:54.009399 396 webserver.cc:460] Webserver started at http://127.0.99.1:38759/ using document root <none> and password file <none>
I20240501 07:59:54.009546 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:54.009594 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:54.009671 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:54.010051 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "7f47cbe8cbcf48c6a7d4f383aa546810"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.011390 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:54.012192 3404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.012398 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:54.012456 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "7f47cbe8cbcf48c6a7d4f383aa546810"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.012518 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:54.042943 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:54.043323 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:54.043599 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:54.044041 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:54.044081 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.044124 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:54.044152 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.047891 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43411
I20240501 07:59:54.047930 3467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43411 every 8 connection(s)
I20240501 07:59:54.048635 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:54.050740 3472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.050799 3473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.051270 3475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:54.051811 396 server_base.cc:1034] running on GCE node
I20240501 07:59:54.051955 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:54.051998 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:54.052019 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550394052019 us; error 0 us; skew 500 ppm
I20240501 07:59:54.052847 3468 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33059
I20240501 07:59:54.052942 396 webserver.cc:460] Webserver started at http://127.0.99.2:34373/ using document root <none> and password file <none>
I20240501 07:59:54.052954 3468 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:54.053131 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:54.053184 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:54.053207 3468 heartbeater.cc:511] Master 127.0.99.62:33059 requested a full tablet report, sending...
I20240501 07:59:54.053251 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:54.053645 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "6cfb475a38b04ad49e7907a718b5ebce"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.053836 3339 ts_manager.cc:194] Registered new tserver with Master: 7f47cbe8cbcf48c6a7d4f383aa546810 (127.0.99.1:43411)
I20240501 07:59:54.054839 3339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33752
I20240501 07:59:54.055300 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:54.056192 3480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.056432 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:54.056490 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "6cfb475a38b04ad49e7907a718b5ebce"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.056557 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:54.064337 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:54.064646 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:54.064911 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:54.065304 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:54.065339 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.065369 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:54.065403 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.069082 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:38483
I20240501 07:59:54.069139 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:38483 every 8 connection(s)
I20240501 07:59:54.069800 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:54.071324 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.071733 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:54.072113 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:54.072571 396 server_base.cc:1034] running on GCE node
I20240501 07:59:54.072700 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 07:59:54.072736 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 07:59:54.072755 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550394072755 us; error 0 us; skew 500 ppm
I20240501 07:59:54.073675 396 webserver.cc:460] Webserver started at http://127.0.99.3:36081/ using document root <none> and password file <none>
I20240501 07:59:54.073807 396 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:54.073817 3544 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33059
I20240501 07:59:54.073853 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:54.073901 3544 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:54.073922 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:54.074095 3544 heartbeater.cc:511] Master 127.0.99.62:33059 requested a full tablet report, sending...
I20240501 07:59:54.074272 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.074671 3339 ts_manager.cc:194] Registered new tserver with Master: 6cfb475a38b04ad49e7907a718b5ebce (127.0.99.2:38483)
I20240501 07:59:54.075155 3339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33758
I20240501 07:59:54.075636 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:54.076473 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.076678 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:54.076740 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7"
format_stamp: "Formatted at 2024-05-01 07:59:54 on dist-test-slave-rp4l"
I20240501 07:59:54.076802 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:54.097549 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:54.097959 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:54.098233 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:54.098702 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:54.098745 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.098788 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:54.098817 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:54.102867 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:46859
I20240501 07:59:54.102962 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:46859 every 8 connection(s)
I20240501 07:59:54.106987 3619 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33059
I20240501 07:59:54.107093 3619 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:54.107298 3619 heartbeater.cc:511] Master 127.0.99.62:33059 requested a full tablet report, sending...
I20240501 07:59:54.107836 3339 ts_manager.cc:194] Registered new tserver with Master: 3213fe2ce0a04e05b66d0a140aa3f4d7 (127.0.99.3:46859)
I20240501 07:59:54.108316 3339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33774
I20240501 07:59:54.108814 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005526779s
I20240501 07:59:55.055716 3468 heartbeater.cc:503] Master 127.0.99.62:33059 was elected leader, sending a full tablet report...
I20240501 07:59:55.075963 3544 heartbeater.cc:503] Master 127.0.99.62:33059 was elected leader, sending a full tablet report...
I20240501 07:59:55.109100 3619 heartbeater.cc:503] Master 127.0.99.62:33059 was elected leader, sending a full tablet report...
I20240501 07:59:55.138479 396 test_util.cc:268] Using random seed: -549424723
I20240501 07:59:55.145349 3339 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33800:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 07:59:55.145782 3339 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 07:59:55.155097 3583 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc026596dba41efa25558a0dc63c62e (DEFAULT_TABLE table=test-workload [id=697ede9d32144723bef0a59f60ed1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:55.155426 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc026596dba41efa25558a0dc63c62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:55.155442 3508 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc026596dba41efa25558a0dc63c62e (DEFAULT_TABLE table=test-workload [id=697ede9d32144723bef0a59f60ed1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:55.155705 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc026596dba41efa25558a0dc63c62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:55.156554 3432 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc026596dba41efa25558a0dc63c62e (DEFAULT_TABLE table=test-workload [id=697ede9d32144723bef0a59f60ed1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:55.156819 3432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc026596dba41efa25558a0dc63c62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:55.157732 3640 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: Bootstrap starting.
I20240501 07:59:55.157897 3641 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: Bootstrap starting.
I20240501 07:59:55.158799 3640 tablet_bootstrap.cc:654] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:55.158799 3641 tablet_bootstrap.cc:654] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:55.159497 3642 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: Bootstrap starting.
I20240501 07:59:55.159983 3641 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: No bootstrap required, opened a new log
I20240501 07:59:55.160074 3641 ts_tablet_manager.cc:1367] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:55.160152 3640 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: No bootstrap required, opened a new log
I20240501 07:59:55.160241 3640 ts_tablet_manager.cc:1367] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:55.160320 3642 tablet_bootstrap.cc:654] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:55.160640 3641 raft_consensus.cc:348] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.160669 3640 raft_consensus.cc:348] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.160748 3641 raft_consensus.cc:374] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:55.160761 3640 raft_consensus.cc:374] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:55.160781 3641 raft_consensus.cc:729] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3213fe2ce0a04e05b66d0a140aa3f4d7, State: Initialized, Role: FOLLOWER
I20240501 07:59:55.160794 3640 raft_consensus.cc:729] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfb475a38b04ad49e7907a718b5ebce, State: Initialized, Role: FOLLOWER
I20240501 07:59:55.160976 3641 consensus_queue.cc:260] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.160986 3640 consensus_queue.cc:260] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.161337 3642 tablet_bootstrap.cc:492] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: No bootstrap required, opened a new log
I20240501 07:59:55.161386 3641 ts_tablet_manager.cc:1398] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:55.161392 3640 ts_tablet_manager.cc:1398] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:55.161453 3642 ts_tablet_manager.cc:1367] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:55.161841 3642 raft_consensus.cc:348] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.161988 3642 raft_consensus.cc:374] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:55.162075 3642 raft_consensus.cc:729] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f47cbe8cbcf48c6a7d4f383aa546810, State: Initialized, Role: FOLLOWER
I20240501 07:59:55.162248 3642 consensus_queue.cc:260] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.162575 3642 ts_tablet_manager.cc:1398] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:55.582443 3648 raft_consensus.cc:482] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:55.582608 3648 raft_consensus.cc:504] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.583231 3648 leader_election.cc:288] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cfb475a38b04ad49e7907a718b5ebce (127.0.99.2:38483), 3213fe2ce0a04e05b66d0a140aa3f4d7 (127.0.99.3:46859)
I20240501 07:59:55.586584 3518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc026596dba41efa25558a0dc63c62e" candidate_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfb475a38b04ad49e7907a718b5ebce" is_pre_election: true
I20240501 07:59:55.586617 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc026596dba41efa25558a0dc63c62e" candidate_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" is_pre_election: true
I20240501 07:59:55.586767 3593 raft_consensus.cc:2454] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f47cbe8cbcf48c6a7d4f383aa546810 in term 0.
I20240501 07:59:55.586768 3518 raft_consensus.cc:2454] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f47cbe8cbcf48c6a7d4f383aa546810 in term 0.
I20240501 07:59:55.587101 3407 leader_election.cc:302] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3213fe2ce0a04e05b66d0a140aa3f4d7, 7f47cbe8cbcf48c6a7d4f383aa546810; no voters:
I20240501 07:59:55.587262 3648 raft_consensus.cc:2781] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:55.587320 3648 raft_consensus.cc:482] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:55.587356 3648 raft_consensus.cc:3037] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:55.588150 3648 raft_consensus.cc:504] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.588454 3648 leader_election.cc:288] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [CANDIDATE]: Term 1 election: Requested vote from peers 6cfb475a38b04ad49e7907a718b5ebce (127.0.99.2:38483), 3213fe2ce0a04e05b66d0a140aa3f4d7 (127.0.99.3:46859)
I20240501 07:59:55.588734 3518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc026596dba41efa25558a0dc63c62e" candidate_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfb475a38b04ad49e7907a718b5ebce"
I20240501 07:59:55.588740 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc026596dba41efa25558a0dc63c62e" candidate_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7"
I20240501 07:59:55.588856 3518 raft_consensus.cc:3037] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:55.588856 3593 raft_consensus.cc:3037] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:55.589831 3518 raft_consensus.cc:2454] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f47cbe8cbcf48c6a7d4f383aa546810 in term 1.
I20240501 07:59:55.589831 3593 raft_consensus.cc:2454] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f47cbe8cbcf48c6a7d4f383aa546810 in term 1.
I20240501 07:59:55.590117 3407 leader_election.cc:302] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3213fe2ce0a04e05b66d0a140aa3f4d7, 7f47cbe8cbcf48c6a7d4f383aa546810; no voters:
I20240501 07:59:55.590252 3648 raft_consensus.cc:2781] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:55.590440 3648 raft_consensus.cc:686] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 1 LEADER]: Becoming Leader. State: Replica: 7f47cbe8cbcf48c6a7d4f383aa546810, State: Running, Role: LEADER
I20240501 07:59:55.590577 3648 consensus_queue.cc:237] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } }
I20240501 07:59:55.592074 3338 catalog_manager.cc:5484] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f47cbe8cbcf48c6a7d4f383aa546810 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f47cbe8cbcf48c6a7d4f383aa546810" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:55.973115 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:55.973388 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 07:59:55.973400 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:55.973500 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:56.047734 3648 consensus_queue.cc:1035] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3213fe2ce0a04e05b66d0a140aa3f4d7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:56.050912 3648 consensus_queue.cc:1035] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfb475a38b04ad49e7907a718b5ebce" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:56.975168 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 07:59:57.973809 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:57.974128 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:57.974190 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:57.976728 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 07:59:58.978381 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 07:59:59.974522 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 07:59:59.974917 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 07:59:59.975003 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 07:59:59.979964 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:00.981453 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:01.975294 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:01.975610 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:00:01.975677 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:01.982995 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:02.984609 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:03.975951 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:03.976255 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:00:03.976321 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:03.986325 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:04.988076 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:05.976647 3393 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:05.976946 3393 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:00:05.977006 3393 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:05.989634 3387 placement_policy_util.cc:407] tablet 6cc026596dba41efa25558a0dc63c62e: detected majority of replicas (2 of 3) at location L0
I20240501 08:00:06.652877 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240501 08:00:06.655835 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:00:06.656034 396 tablet_replica.cc:330] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810: stopping tablet replica
I20240501 08:00:06.656142 396 raft_consensus.cc:2229] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:00:06.656345 396 raft_consensus.cc:2258] T 6cc026596dba41efa25558a0dc63c62e P 7f47cbe8cbcf48c6a7d4f383aa546810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.668685 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240501 08:00:06.670397 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240501 08:00:06.673301 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:00:06.673454 396 tablet_replica.cc:330] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce: stopping tablet replica
I20240501 08:00:06.673548 396 raft_consensus.cc:2229] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.673664 396 raft_consensus.cc:2258] T 6cc026596dba41efa25558a0dc63c62e P 6cfb475a38b04ad49e7907a718b5ebce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.685686 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240501 08:00:06.687363 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240501 08:00:06.690472 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 08:00:06.690649 396 tablet_replica.cc:330] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7: stopping tablet replica
I20240501 08:00:06.690748 396 raft_consensus.cc:2229] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.690838 396 raft_consensus.cc:2258] T 6cc026596dba41efa25558a0dc63c62e P 3213fe2ce0a04e05b66d0a140aa3f4d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.702828 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240501 08:00:06.704536 396 master.cc:537] Master@127.0.99.62:33059 shutting down...
I20240501 08:00:06.707199 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:00:06.707340 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.707399 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d783c0bf3fb0443bbd741ed1bc4154eb: stopping tablet replica
I20240501 08:00:06.719401 396 master.cc:552] Master@127.0.99.62:33059 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12783 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 08:00:06.725185 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:34785
I20240501 08:00:06.725450 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:06.727569 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.728142 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.728252 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:06.728755 396 server_base.cc:1034] running on GCE node
I20240501 08:00:06.728871 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:06.728902 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:06.728914 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550406728914 us; error 0 us; skew 500 ppm
I20240501 08:00:06.729743 396 webserver.cc:460] Webserver started at http://127.0.99.62:34551/ using document root <none> and password file <none>
I20240501 08:00:06.729858 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:06.729894 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:06.729944 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:06.730266 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/master-0-root/instance:
uuid: "47dfd47c09df4bfdabe6c0171bd97805"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.731586 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 08:00:06.732337 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.732581 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:06.732645 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/master-0-root
uuid: "47dfd47c09df4bfdabe6c0171bd97805"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.732708 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:06.756948 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:06.757301 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:06.761175 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:34785
I20240501 08:00:06.762596 3722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:34785 every 8 connection(s)
I20240501 08:00:06.763041 3723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:06.764712 3723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805: Bootstrap starting.
I20240501 08:00:06.765406 3723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:06.766292 3723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805: No bootstrap required, opened a new log
I20240501 08:00:06.766680 3723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER }
I20240501 08:00:06.766767 3723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:06.766794 3723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47dfd47c09df4bfdabe6c0171bd97805, State: Initialized, Role: FOLLOWER
I20240501 08:00:06.766917 3723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER }
I20240501 08:00:06.766980 3723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:00:06.767017 3723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:00:06.767061 3723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:06.767676 3723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER }
I20240501 08:00:06.767791 3723 leader_election.cc:302] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47dfd47c09df4bfdabe6c0171bd97805; no voters:
I20240501 08:00:06.767946 3723 leader_election.cc:288] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:00:06.768039 3726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:06.768213 3726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [term 1 LEADER]: Becoming Leader. State: Replica: 47dfd47c09df4bfdabe6c0171bd97805, State: Running, Role: LEADER
I20240501 08:00:06.768330 3723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:00:06.768325 3726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER }
I20240501 08:00:06.768730 3727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47dfd47c09df4bfdabe6c0171bd97805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER } }
I20240501 08:00:06.768757 3728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47dfd47c09df4bfdabe6c0171bd97805. Latest consensus state: current_term: 1 leader_uuid: "47dfd47c09df4bfdabe6c0171bd97805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dfd47c09df4bfdabe6c0171bd97805" member_type: VOTER } }
I20240501 08:00:06.768815 3727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [sys.catalog]: This master's current role is: LEADER
I20240501 08:00:06.768836 3728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805 [sys.catalog]: This master's current role is: LEADER
I20240501 08:00:06.769060 3731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:00:06.769788 3731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:00:06.770000 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:00:06.771401 3731 catalog_manager.cc:1347] Generated new cluster ID: 790ffdc351284871a9b9f905a0cde77d
I20240501 08:00:06.771445 3731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:00:06.777643 3731 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:00:06.778123 3731 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:00:06.785820 3731 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 47dfd47c09df4bfdabe6c0171bd97805: Generated new TSK 0
I20240501 08:00:06.785953 3731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:00:06.802218 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:06.804059 3745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.804716 3747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.805255 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:06.805742 396 server_base.cc:1034] running on GCE node
I20240501 08:00:06.805881 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:06.805918 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:06.805932 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550406805932 us; error 0 us; skew 500 ppm
I20240501 08:00:06.806775 396 webserver.cc:460] Webserver started at http://127.0.99.1:34947/ using document root <none> and password file <none>
I20240501 08:00:06.806910 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:06.806952 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:06.807022 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:06.807363 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/instance:
uuid: "504c682c767543748142f114ffa7be2f"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.808642 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:06.809381 3752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.809582 396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 08:00:06.809648 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root
uuid: "504c682c767543748142f114ffa7be2f"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.809708 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:06.818132 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:06.818404 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:06.818671 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:06.819080 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:06.819116 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.819154 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:06.819180 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.822836 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:39565
I20240501 08:00:06.822870 3815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:39565 every 8 connection(s)
I20240501 08:00:06.823565 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:06.825057 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.825528 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.825984 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:06.826457 396 server_base.cc:1034] running on GCE node
I20240501 08:00:06.826642 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:06.826686 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:06.826714 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550406826712 us; error 0 us; skew 500 ppm
I20240501 08:00:06.827662 396 webserver.cc:460] Webserver started at http://127.0.99.2:37803/ using document root <none> and password file <none>
I20240501 08:00:06.827811 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:06.827863 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:06.827988 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:06.828387 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/instance:
uuid: "d5c9f893d7344b4f841e68668e19ab97"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.828387 3816 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:06.828528 3816 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:06.828725 3816 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:06.829324 3687 ts_manager.cc:194] Registered new tserver with Master: 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:06.829816 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:06.829977 3687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41172
I20240501 08:00:06.830724 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.830936 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:06.831000 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root
uuid: "d5c9f893d7344b4f841e68668e19ab97"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.831066 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:06.860843 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:06.861222 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:06.861531 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:06.861990 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:06.862030 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.862072 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:06.862099 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.865940 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:34681
I20240501 08:00:06.865968 3891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:34681 every 8 connection(s)
I20240501 08:00:06.866688 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:06.868831 3896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.868992 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:06.869514 3898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:06.869994 396 server_base.cc:1034] running on GCE node
I20240501 08:00:06.870143 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:06.870190 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:06.870209 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550406870209 us; error 0 us; skew 500 ppm
I20240501 08:00:06.870747 3892 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:06.870836 3892 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:06.871039 3892 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:06.871100 396 webserver.cc:460] Webserver started at http://127.0.99.3:42639/ using document root <none> and password file <none>
I20240501 08:00:06.871261 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:06.871302 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:06.871354 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:06.871606 3687 ts_manager.cc:194] Registered new tserver with Master: d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:06.871745 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/instance:
uuid: "f818705fd267423dba3ae9ae51d3a8a1"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.872108 3687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41182
I20240501 08:00:06.873111 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 08:00:06.873917 3903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.874117 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:06.874176 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root
uuid: "f818705fd267423dba3ae9ae51d3a8a1"
format_stamp: "Formatted at 2024-05-01 08:00:06 on dist-test-slave-rp4l"
I20240501 08:00:06.874239 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:06.887171 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:06.887473 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:06.887737 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:06.888136 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:06.888171 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.888208 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:06.888235 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:06.892391 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:39165
I20240501 08:00:06.892439 3966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:39165 every 8 connection(s)
I20240501 08:00:06.896350 3967 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:06.896435 3967 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:06.896615 3967 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:06.897127 3687 ts_manager.cc:194] Registered new tserver with Master: f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:06.897176 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004429862s
I20240501 08:00:06.897620 3687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41188
I20240501 08:00:07.830823 3816 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:07.872972 3892 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:07.898384 3967 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:07.922668 396 test_util.cc:268] Using random seed: -536640535
I20240501 08:00:07.929899 3687 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41204:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 08:00:07.930476 3687 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:00:07.943110 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea8a83bd3a9408e877cf926ce6bf466 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:00:07.943297 3930 tablet_service.cc:1467] Processing CreateTablet for tablet b909fdab57ca42a0b05c41dde754743a (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:00:07.943437 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea8a83bd3a9408e877cf926ce6bf466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.943661 3929 tablet_service.cc:1467] Processing CreateTablet for tablet 728c3dff60a8498e99bc434ff175083f (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:00:07.943722 3855 tablet_service.cc:1467] Processing CreateTablet for tablet b909fdab57ca42a0b05c41dde754743a (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:00:07.943879 3929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728c3dff60a8498e99bc434ff175083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.943938 3855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909fdab57ca42a0b05c41dde754743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.945662 3987 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.945861 3855 tablet_service.cc:1467] Processing CreateTablet for tablet f1ef543279a8493295c4439e6d8d87cd (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:00:07.945964 3929 tablet_service.cc:1467] Processing CreateTablet for tablet f1ef543279a8493295c4439e6d8d87cd (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:00:07.946092 3855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ef543279a8493295c4439e6d8d87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.946192 3929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ef543279a8493295c4439e6d8d87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.946831 3854 tablet_service.cc:1467] Processing CreateTablet for tablet 728c3dff60a8498e99bc434ff175083f (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:00:07.946852 3928 tablet_service.cc:1467] Processing CreateTablet for tablet 9d827adaf117470fa9fada0d8f245671 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:00:07.946993 3987 tablet_bootstrap.cc:654] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.947083 3928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d827adaf117470fa9fada0d8f245671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.947023 3923 tablet_service.cc:1467] Processing CreateTablet for tablet a4711451691345009da4d87f91300ca1 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:00:07.947232 3923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4711451691345009da4d87f91300ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.946852 3922 tablet_service.cc:1467] Processing CreateTablet for tablet 824019b3e8674c329bdeb8467db1f6ff (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:00:07.947485 3922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824019b3e8674c329bdeb8467db1f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.947697 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7085039b984324870248db803ece14 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:00:07.948050 3987 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.948073 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:00:07.948132 3987 ts_tablet_manager.cc:1367] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:07.948450 3849 tablet_service.cc:1467] Processing CreateTablet for tablet e1c9c65f0c0a4a53aafc7a04061dc909 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:00:07.948545 3987 raft_consensus.cc:348] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.948638 3987 raft_consensus.cc:374] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.948669 3987 raft_consensus.cc:729] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.948789 3987 consensus_queue.cc:260] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.948815 3848 tablet_service.cc:1467] Processing CreateTablet for tablet a4711451691345009da4d87f91300ca1 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:00:07.949124 3987 ts_tablet_manager.cc:1398] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.947430 3852 tablet_service.cc:1467] Processing CreateTablet for tablet 80c853e31bc74cb38778ed1ab133d966 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:00:07.949172 3847 tablet_service.cc:1467] Processing CreateTablet for tablet 824019b3e8674c329bdeb8467db1f6ff (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:00:07.949219 3924 tablet_service.cc:1467] Processing CreateTablet for tablet e1c9c65f0c0a4a53aafc7a04061dc909 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:00:07.949425 3924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c9c65f0c0a4a53aafc7a04061dc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.947237 3853 tablet_service.cc:1467] Processing CreateTablet for tablet 9d827adaf117470fa9fada0d8f245671 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:00:07.947037 3854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728c3dff60a8498e99bc434ff175083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.949870 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c9c65f0c0a4a53aafc7a04061dc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.949980 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7085039b984324870248db803ece14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.950711 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.950843 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c853e31bc74cb38778ed1ab133d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.951273 3925 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:00:07.951489 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.951766 3853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d827adaf117470fa9fada0d8f245671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.952492 3847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824019b3e8674c329bdeb8467db1f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.953348 3926 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7085039b984324870248db803ece14 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:00:07.953562 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7085039b984324870248db803ece14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.954173 3848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4711451691345009da4d87f91300ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.955117 3927 tablet_service.cc:1467] Processing CreateTablet for tablet 80c853e31bc74cb38778ed1ab133d966 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:00:07.955325 3927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c853e31bc74cb38778ed1ab133d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.957226 3851 tablet_service.cc:1467] Processing CreateTablet for tablet dc118979132748e1a45e3e33e655ac11 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:00:07.957336 3988 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.957435 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc118979132748e1a45e3e33e655ac11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.943110 3856 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea8a83bd3a9408e877cf926ce6bf466 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:00:07.958441 3988 tablet_bootstrap.cc:654] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.958515 3856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea8a83bd3a9408e877cf926ce6bf466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.958870 3927 tablet_service.cc:1467] Processing CreateTablet for tablet dc118979132748e1a45e3e33e655ac11 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:00:07.959060 3927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc118979132748e1a45e3e33e655ac11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.959203 3930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909fdab57ca42a0b05c41dde754743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.953567 3987 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.960114 3987 tablet_bootstrap.cc:654] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.961139 3988 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.961228 3988 ts_tablet_manager.cc:1367] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:00:07.961519 3987 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.961601 3987 ts_tablet_manager.cc:1367] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 08:00:07.961633 3988 raft_consensus.cc:348] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:07.961755 3988 raft_consensus.cc:374] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.961796 3988 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.961912 3988 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:07.961977 3987 raft_consensus.cc:348] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.962057 3987 raft_consensus.cc:374] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.962096 3987 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.962270 3988 ts_tablet_manager.cc:1398] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.962234 3987 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.962606 3987 ts_tablet_manager.cc:1398] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.962684 3988 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.962850 3987 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.963550 3988 tablet_bootstrap.cc:654] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.963580 3987 tablet_bootstrap.cc:654] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.964859 3987 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.964978 3987 ts_tablet_manager.cc:1367] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:00:07.965397 3987 raft_consensus.cc:348] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.965498 3988 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.965564 3987 raft_consensus.cc:374] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.965668 3987 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.965672 3988 ts_tablet_manager.cc:1367] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:07.965891 3987 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.966172 3988 raft_consensus.cc:348] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.966305 3988 raft_consensus.cc:374] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.966363 3988 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.966503 3988 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.966650 3780 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea8a83bd3a9408e877cf926ce6bf466 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 08:00:07.966765 3778 tablet_service.cc:1467] Processing CreateTablet for tablet 728c3dff60a8498e99bc434ff175083f (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 08:00:07.966881 3780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea8a83bd3a9408e877cf926ce6bf466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.966987 3778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728c3dff60a8498e99bc434ff175083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.967268 3779 tablet_service.cc:1467] Processing CreateTablet for tablet b909fdab57ca42a0b05c41dde754743a (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 08:00:07.967384 3988 ts_tablet_manager.cc:1398] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:07.967484 3779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909fdab57ca42a0b05c41dde754743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.967542 3988 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.968374 3988 tablet_bootstrap.cc:654] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.968964 3780 tablet_service.cc:1467] Processing CreateTablet for tablet dc118979132748e1a45e3e33e655ac11 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 08:00:07.968967 3778 tablet_service.cc:1467] Processing CreateTablet for tablet f1ef543279a8493295c4439e6d8d87cd (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 08:00:07.969218 3780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc118979132748e1a45e3e33e655ac11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.969282 3777 tablet_service.cc:1467] Processing CreateTablet for tablet 9d827adaf117470fa9fada0d8f245671 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 08:00:07.969348 3778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ef543279a8493295c4439e6d8d87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.969458 3777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d827adaf117470fa9fada0d8f245671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.970468 3776 tablet_service.cc:1467] Processing CreateTablet for tablet 80c853e31bc74cb38778ed1ab133d966 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 08:00:07.970839 3775 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7085039b984324870248db803ece14 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 08:00:07.971073 3775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7085039b984324870248db803ece14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.972402 3774 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 08:00:07.972577 3774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.973897 3773 tablet_service.cc:1467] Processing CreateTablet for tablet e1c9c65f0c0a4a53aafc7a04061dc909 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 08:00:07.974094 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c9c65f0c0a4a53aafc7a04061dc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.974908 3771 tablet_service.cc:1467] Processing CreateTablet for tablet 824019b3e8674c329bdeb8467db1f6ff (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 08:00:07.975121 3771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824019b3e8674c329bdeb8467db1f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.976527 3772 tablet_service.cc:1467] Processing CreateTablet for tablet a4711451691345009da4d87f91300ca1 (DEFAULT_TABLE table=test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 08:00:07.976716 3772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4711451691345009da4d87f91300ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.976897 3776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c853e31bc74cb38778ed1ab133d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.966189 3987 ts_tablet_manager.cc:1398] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:07.978873 3987 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.979704 3996 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:07.979735 3987 tablet_bootstrap.cc:654] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.980568 3996 tablet_bootstrap.cc:654] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.980733 3987 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.980809 3987 ts_tablet_manager.cc:1367] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:07.981128 3987 raft_consensus.cc:348] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.981283 3987 raft_consensus.cc:374] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.981334 3987 raft_consensus.cc:729] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.981499 3988 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.981473 3987 consensus_queue.cc:260] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.981551 3996 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:07.981575 3988 ts_tablet_manager.cc:1367] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 08:00:07.981666 3996 ts_tablet_manager.cc:1367] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:07.981815 3987 ts_tablet_manager.cc:1398] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:07.981956 3987 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.982009 3988 raft_consensus.cc:348] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.982082 3988 raft_consensus.cc:374] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.982060 3996 raft_consensus.cc:348] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:07.982111 3988 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.982172 3996 raft_consensus.cc:374] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.982208 3996 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.982275 3988 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.982360 3996 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:07.982513 3988 ts_tablet_manager.cc:1398] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.982687 3988 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.982723 3996 ts_tablet_manager.cc:1398] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.983494 3988 tablet_bootstrap.cc:654] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.983675 3996 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:07.984529 3987 tablet_bootstrap.cc:654] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.984614 3996 tablet_bootstrap.cc:654] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.985569 3996 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:07.985642 3996 ts_tablet_manager.cc:1367] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:07.985695 3987 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.985785 3987 ts_tablet_manager.cc:1367] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:00:07.985997 3996 raft_consensus.cc:348] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.986079 3996 raft_consensus.cc:374] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.986107 3996 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.986172 3987 raft_consensus.cc:348] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.986199 3996 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.986459 3996 ts_tablet_manager.cc:1398] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.986588 3988 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.986680 3987 raft_consensus.cc:374] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.986747 3988 ts_tablet_manager.cc:1367] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:07.986747 3987 raft_consensus.cc:729] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.986914 3987 consensus_queue.cc:260] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.987177 3988 raft_consensus.cc:348] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.987262 3988 raft_consensus.cc:374] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.987291 3988 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.987386 3988 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.987610 3988 ts_tablet_manager.cc:1398] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:07.986650 3996 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:07.987797 3988 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.988509 3996 tablet_bootstrap.cc:654] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.988639 3987 ts_tablet_manager.cc:1398] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:00:07.988785 3987 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.988509 3988 tablet_bootstrap.cc:654] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.989567 3987 tablet_bootstrap.cc:654] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.990550 3987 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.990648 3987 ts_tablet_manager.cc:1367] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:00:07.990927 3996 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:07.991003 3996 ts_tablet_manager.cc:1367] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:07.991015 3987 raft_consensus.cc:348] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.991086 3988 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.991112 3987 raft_consensus.cc:374] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.991147 3988 ts_tablet_manager.cc:1367] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:00:07.991189 3987 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.991338 3987 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.991369 3996 raft_consensus.cc:348] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.991519 3996 raft_consensus.cc:374] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.991575 3996 raft_consensus.cc:729] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.991560 3988 raft_consensus.cc:348] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.991678 3988 raft_consensus.cc:374] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.991689 3987 ts_tablet_manager.cc:1398] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.991787 3988 raft_consensus.cc:729] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.991734 3996 consensus_queue.cc:260] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.991948 3987 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.991961 3988 consensus_queue.cc:260] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.992174 3996 ts_tablet_manager.cc:1398] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.992362 3996 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:07.992805 3987 tablet_bootstrap.cc:654] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.993186 3996 tablet_bootstrap.cc:654] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.993420 3988 ts_tablet_manager.cc:1398] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:07.994020 3988 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.994185 3996 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:07.994287 3996 ts_tablet_manager.cc:1367] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:07.994707 3996 raft_consensus.cc:348] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.994813 3996 raft_consensus.cc:374] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.994843 3996 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.994974 3996 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.995052 3988 tablet_bootstrap.cc:654] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.995270 3996 ts_tablet_manager.cc:1398] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.995461 3996 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:07.995874 3987 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:07.995949 3987 ts_tablet_manager.cc:1367] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:07.996273 3996 tablet_bootstrap.cc:654] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.996322 3987 raft_consensus.cc:348] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.996420 3987 raft_consensus.cc:374] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.996460 3987 raft_consensus.cc:729] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.996563 3987 consensus_queue.cc:260] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.996806 3987 ts_tablet_manager.cc:1398] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:07.997210 3988 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:07.997287 3988 ts_tablet_manager.cc:1367] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:00:07.997354 3996 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:07.997421 3996 ts_tablet_manager.cc:1367] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:07.997659 3988 raft_consensus.cc:348] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.997740 3988 raft_consensus.cc:374] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.997759 3996 raft_consensus.cc:348] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.997769 3988 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.997896 3996 raft_consensus.cc:374] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:07.997929 3996 raft_consensus.cc:729] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.997961 3987 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:07.998003 3988 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.998013 3996 consensus_queue.cc:260] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:07.998272 3988 ts_tablet_manager.cc:1398] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:07.998301 3996 ts_tablet_manager.cc:1398] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:07.998418 3988 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:07.999022 3987 tablet_bootstrap.cc:654] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.999347 3988 tablet_bootstrap.cc:654] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:07.998418 3996 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.000308 3988 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.000387 3988 ts_tablet_manager.cc:1367] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.000629 3996 tablet_bootstrap.cc:654] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.000775 3988 raft_consensus.cc:348] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.000867 3988 raft_consensus.cc:374] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.000896 3988 raft_consensus.cc:729] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.001019 3988 consensus_queue.cc:260] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.001329 3988 ts_tablet_manager.cc:1398] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.001497 3988 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:08.002305 3988 tablet_bootstrap.cc:654] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.002914 3987 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:08.002985 3987 ts_tablet_manager.cc:1367] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:00:08.003252 3988 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.003324 3988 ts_tablet_manager.cc:1367] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.003355 3987 raft_consensus.cc:348] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.003434 3987 raft_consensus.cc:374] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.003460 3987 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.003556 3987 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.003679 3988 raft_consensus.cc:348] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.003748 3988 raft_consensus.cc:374] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.003777 3988 raft_consensus.cc:729] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.003822 3987 ts_tablet_manager.cc:1398] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.003881 3988 consensus_queue.cc:260] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.003960 3987 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:08.004110 3988 ts_tablet_manager.cc:1398] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.004241 3988 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:08.004770 3987 tablet_bootstrap.cc:654] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.005005 3988 tablet_bootstrap.cc:654] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.005900 3988 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.005884 3987 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:08.006016 3988 ts_tablet_manager.cc:1367] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.006021 3987 ts_tablet_manager.cc:1367] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:08.006459 3987 raft_consensus.cc:348] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.006443 3988 raft_consensus.cc:348] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.006601 3987 raft_consensus.cc:374] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.006666 3987 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.006664 3988 raft_consensus.cc:374] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.006765 3988 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.006783 3987 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.006881 3988 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.006976 3987 ts_tablet_manager.cc:1398] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.007117 3988 ts_tablet_manager.cc:1398] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.007300 3988 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:08.007115 3987 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:08.008080 3988 tablet_bootstrap.cc:654] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.008319 3987 tablet_bootstrap.cc:654] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.008450 4000 raft_consensus.cc:482] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.008533 4000 raft_consensus.cc:504] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.008930 4000 leader_election.cc:288] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.009248 3996 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.009292 3987 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:08.009322 3996 ts_tablet_manager.cc:1367] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 08:00:08.009361 3987 ts_tablet_manager.cc:1367] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.009722 3987 raft_consensus.cc:348] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.009814 3987 raft_consensus.cc:374] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.009845 3987 raft_consensus.cc:729] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.010083 3996 raft_consensus.cc:348] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.010165 3996 raft_consensus.cc:374] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.010197 3996 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.010293 3996 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.010540 3996 ts_tablet_manager.cc:1398] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.011047 3988 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.011085 3996 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.011134 3988 ts_tablet_manager.cc:1367] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:08.011509 3988 raft_consensus.cc:348] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.011600 3988 raft_consensus.cc:374] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.011629 3988 raft_consensus.cc:729] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.011720 3988 consensus_queue.cc:260] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.011901 3996 tablet_bootstrap.cc:654] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.011957 3988 ts_tablet_manager.cc:1398] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.012121 3988 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
I20240501 08:00:08.012396 3987 consensus_queue.cc:260] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.012660 3987 ts_tablet_manager.cc:1398] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:08.012830 3987 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:08.013478 3988 tablet_bootstrap.cc:654] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.013653 3996 tablet_bootstrap.cc:492] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.013733 3996 ts_tablet_manager.cc:1367] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:00:08.013887 3990 raft_consensus.cc:482] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.013968 3990 raft_consensus.cc:504] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.014082 3996 raft_consensus.cc:348] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.014168 3996 raft_consensus.cc:374] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.014194 3996 raft_consensus.cc:729] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.014433 3990 leader_election.cc:288] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.013654 3987 tablet_bootstrap.cc:654] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.014411 3996 consensus_queue.cc:260] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.015429 3996 ts_tablet_manager.cc:1398] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.015594 3996 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.016417 3996 tablet_bootstrap.cc:654] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.017315 3996 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.017397 3996 ts_tablet_manager.cc:1367] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.017655 3988 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.017717 3988 ts_tablet_manager.cc:1367] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:00:08.017783 3996 raft_consensus.cc:348] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.017863 3996 raft_consensus.cc:374] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.017890 3996 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.017998 3996 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.018051 3988 raft_consensus.cc:348] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.018124 3988 raft_consensus.cc:374] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.018151 3988 raft_consensus.cc:729] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.018232 3996 ts_tablet_manager.cc:1398] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.018245 3988 consensus_queue.cc:260] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.018368 3996 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.018362 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.018460 3988 ts_tablet_manager.cc:1398] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.018591 3988 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Bootstrap starting.
W20240501 08:00:08.018728 3753 leader_election.cc:341] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:00:08.019209 3996 tablet_bootstrap.cc:654] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.019361 3988 tablet_bootstrap.cc:654] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.021085 3996 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.021157 3996 ts_tablet_manager.cc:1367] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:08.021529 3996 raft_consensus.cc:348] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.021610 3996 raft_consensus.cc:374] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.021638 3996 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.021739 3996 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.021780 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.021889 3790 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.021905 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.021981 3996 ts_tablet_manager.cc:1398] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.022117 3996 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.022221 3987 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:08.022255 3831 leader_election.cc:302] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.022293 3987 ts_tablet_manager.cc:1367] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 08:00:08.022405 3990 raft_consensus.cc:2781] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 08:00:08.022444 3753 leader_election.cc:341] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:00:08.022521 3753 leader_election.cc:302] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 504c682c767543748142f114ffa7be2f; no voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1
I20240501 08:00:08.022666 3987 raft_consensus.cc:348] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.022763 3987 raft_consensus.cc:374] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.022794 3987 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.022450 3990 raft_consensus.cc:482] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.022861 3988 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: No bootstrap required, opened a new log
I20240501 08:00:08.022881 3990 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.022902 3996 tablet_bootstrap.cc:654] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.022917 3987 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.023267 3987 ts_tablet_manager.cc:1398] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.022922 3988 ts_tablet_manager.cc:1367] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:00:08.023666 3988 raft_consensus.cc:348] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.023731 4000 raft_consensus.cc:2726] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:00:08.023757 3988 raft_consensus.cc:374] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.023777 3990 raft_consensus.cc:504] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.024099 3990 leader_election.cc:288] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.024498 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.024595 3790 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.025429 3790 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.025732 3987 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: Bootstrap starting.
I20240501 08:00:08.026072 3831 leader_election.cc:302] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.026216 3990 raft_consensus.cc:2781] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.023787 3988 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.026386 3990 raft_consensus.cc:686] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:08.026392 3988 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.026527 3990 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.026729 3988 ts_tablet_manager.cc:1398] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:08.027168 3987 tablet_bootstrap.cc:654] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.027456 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.027556 3941 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.027920 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.028008 3940 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.028298 3987 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: No bootstrap required, opened a new log
I20240501 08:00:08.028378 3987 ts_tablet_manager.cc:1367] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:08.028730 3987 raft_consensus.cc:348] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:08.028803 3940 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.028827 3987 raft_consensus.cc:374] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.028867 3987 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.028975 3987 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:08.029247 3987 ts_tablet_manager.cc:1398] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.029405 3686 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.030123 3996 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.030205 3996 ts_tablet_manager.cc:1367] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 08:00:08.030515 3996 raft_consensus.cc:348] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.030608 3996 raft_consensus.cc:374] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.030668 3996 raft_consensus.cc:729] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.030776 3996 consensus_queue.cc:260] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.031008 3996 ts_tablet_manager.cc:1398] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.031180 3996 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.032006 3996 tablet_bootstrap.cc:654] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.032981 3996 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.033066 3996 ts_tablet_manager.cc:1367] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:00:08.033448 3996 raft_consensus.cc:348] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.033524 3996 raft_consensus.cc:374] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.033545 3996 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.033630 3996 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.033843 3996 ts_tablet_manager.cc:1398] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.034021 3996 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f: Bootstrap starting.
I20240501 08:00:08.034741 3996 tablet_bootstrap.cc:654] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f: Neither blocks nor log segments found. Creating new log.
I20240501 08:00:08.035562 3996 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f: No bootstrap required, opened a new log
I20240501 08:00:08.035635 3996 ts_tablet_manager.cc:1367] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:00:08.035980 3996 raft_consensus.cc:348] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.036058 3996 raft_consensus.cc:374] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:00:08.036077 3996 raft_consensus.cc:729] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.036166 3996 consensus_queue.cc:260] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.036374 3996 ts_tablet_manager.cc:1398] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.040462 4033 raft_consensus.cc:482] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.040536 4033 raft_consensus.cc:504] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.040835 4033 leader_election.cc:288] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.041112 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.041113 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.041217 3940 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.041246 3790 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.041451 3829 leader_election.cc:302] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.041643 4033 raft_consensus.cc:2781] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.041684 4033 raft_consensus.cc:482] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.041709 4033 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.042412 4033 raft_consensus.cc:504] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.042719 4033 leader_election.cc:288] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.042934 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.042959 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.043041 3790 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.043056 3940 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.043694 3790 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.043922 3940 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.043941 3831 leader_election.cc:302] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.044154 4033 raft_consensus.cc:2781] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.044230 4033 raft_consensus.cc:686] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:08.044349 4033 consensus_queue.cc:237] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.045187 3993 raft_consensus.cc:482] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.045272 3993 raft_consensus.cc:504] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.045707 3993 leader_election.cc:288] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.045877 3685 catalog_manager.cc:5484] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.049371 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ef543279a8493295c4439e6d8d87cd" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.049480 3790 raft_consensus.cc:2454] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.049520 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ef543279a8493295c4439e6d8d87cd" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.049628 3866 raft_consensus.cc:2454] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.049703 3906 leader_election.cc:302] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.049871 3993 raft_consensus.cc:2781] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.049921 3993 raft_consensus.cc:482] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.049945 3993 raft_consensus.cc:3037] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.050729 3993 raft_consensus.cc:504] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.051010 3993 leader_election.cc:288] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.051254 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ef543279a8493295c4439e6d8d87cd" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.051254 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ef543279a8493295c4439e6d8d87cd" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.051376 3790 raft_consensus.cc:3037] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.051376 3866 raft_consensus.cc:3037] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.052191 3790 raft_consensus.cc:2454] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.052191 3866 raft_consensus.cc:2454] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.052443 3906 leader_election.cc:302] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.052572 3993 raft_consensus.cc:2781] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.052768 3993 raft_consensus.cc:686] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.052886 3993 consensus_queue.cc:237] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.054266 3685 catalog_manager.cc:5484] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:08.078649 3993 raft_consensus.cc:482] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.078779 3993 raft_consensus.cc:504] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.079164 3993 leader_election.cc:288] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.079509 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.079524 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.079623 3790 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.079663 3866 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.079865 3906 leader_election.cc:302] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.080025 3993 raft_consensus.cc:2781] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.080080 3993 raft_consensus.cc:482] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.080104 3993 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.080842 3993 raft_consensus.cc:504] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.081125 3993 leader_election.cc:288] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.081274 3993 raft_consensus.cc:482] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.081393 3993 raft_consensus.cc:504] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.081390 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.081390 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.081508 3790 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.081511 3866 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.081699 3993 leader_election.cc:288] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.081957 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d827adaf117470fa9fada0d8f245671" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.082068 3789 raft_consensus.cc:2454] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.082065 3865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d827adaf117470fa9fada0d8f245671" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.082199 3865 raft_consensus.cc:2454] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.082329 3906 leader_election.cc:302] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.082401 3866 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.082461 3993 raft_consensus.cc:2781] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.082595 3993 raft_consensus.cc:482] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.082674 3993 raft_consensus.cc:3037] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.082607 3904 leader_election.cc:302] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.082839 4041 raft_consensus.cc:2781] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.082899 3790 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.083065 4041 raft_consensus.cc:686] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.083242 4041 consensus_queue.cc:237] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.083436 3993 raft_consensus.cc:504] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.083717 3993 leader_election.cc:288] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.083884 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d827adaf117470fa9fada0d8f245671" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.083971 3790 raft_consensus.cc:3037] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.084625 3790 raft_consensus.cc:2454] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.084803 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d827adaf117470fa9fada0d8f245671" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.084863 3906 leader_election.cc:302] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.084894 3866 raft_consensus.cc:3037] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.084853 3685 catalog_manager.cc:5484] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:08.085354 4041 raft_consensus.cc:2781] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.085496 4041 raft_consensus.cc:686] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.085530 3866 raft_consensus.cc:2454] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.085611 4041 consensus_queue.cc:237] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.086604 4030 raft_consensus.cc:482] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.086700 4030 raft_consensus.cc:504] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.087014 4030 leader_election.cc:288] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.087288 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.087330 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.087386 3940 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 0.
I20240501 08:00:08.087419 3866 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 0.
I20240501 08:00:08.087603 3753 leader_election.cc:302] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.087828 4030 raft_consensus.cc:2781] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.087785 3685 catalog_manager.cc:5484] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:08.087872 4030 raft_consensus.cc:482] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.087908 4030 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.088603 4030 raft_consensus.cc:504] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.088887 4030 leader_election.cc:288] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.089138 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.089234 3866 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.089298 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.089377 3940 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.089928 3866 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 1.
I20240501 08:00:08.090080 3940 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 1.
I20240501 08:00:08.090166 3753 leader_election.cc:302] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.090389 4030 raft_consensus.cc:2781] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.090463 4030 raft_consensus.cc:686] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:08.090570 4030 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.091878 3685 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.094501 4041 raft_consensus.cc:482] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.094569 4041 raft_consensus.cc:504] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.094853 4041 leader_election.cc:288] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.095103 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.095188 3790 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.095387 3906 leader_election.cc:302] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.095413 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.095511 4041 raft_consensus.cc:2781] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.095512 3866 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.095574 4041 raft_consensus.cc:482] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.095603 4041 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.096228 4041 raft_consensus.cc:504] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.096503 4041 leader_election.cc:288] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.096720 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.096786 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.096799 3790 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.096894 3866 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.097517 3790 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.097563 3866 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.097733 3906 leader_election.cc:302] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.097946 4041 raft_consensus.cc:2781] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.098003 4041 raft_consensus.cc:686] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.098105 4041 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.099368 3685 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:08.118285 4030 raft_consensus.cc:482] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.118407 4030 raft_consensus.cc:504] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.118809 4030 leader_election.cc:288] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.119168 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.119168 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.119328 3866 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 0.
I20240501 08:00:08.119325 3940 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 0.
I20240501 08:00:08.119589 3753 leader_election.cc:302] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.119724 4030 raft_consensus.cc:2781] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.119786 4030 raft_consensus.cc:482] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.119832 4030 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.120509 4030 raft_consensus.cc:504] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.120779 4030 leader_election.cc:288] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.121049 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.121122 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.121153 3866 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.121201 3940 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.121922 3940 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 1.
I20240501 08:00:08.121922 3866 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 1.
I20240501 08:00:08.122200 3753 leader_election.cc:302] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.122335 4030 raft_consensus.cc:2781] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.122414 4030 raft_consensus.cc:686] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:08.122532 4030 consensus_queue.cc:237] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.124020 3685 catalog_manager.cc:5484] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.133649 4041 raft_consensus.cc:482] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.133744 4041 raft_consensus.cc:504] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.134040 4041 leader_election.cc:288] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.134286 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.134316 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.134382 3790 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.134418 3866 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.134588 3906 leader_election.cc:302] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.134743 4041 raft_consensus.cc:2781] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.134806 4041 raft_consensus.cc:482] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.134845 4041 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.135488 4041 raft_consensus.cc:504] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.135747 4041 leader_election.cc:288] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:08.136003 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.136014 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.136101 3790 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.136101 3866 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.136703 3866 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.136708 3790 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.136930 3904 leader_election.cc:302] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.137068 4041 raft_consensus.cc:2781] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.137151 4041 raft_consensus.cc:686] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.137291 4041 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.138684 3685 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:08.144260 4033 raft_consensus.cc:482] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.144344 4033 raft_consensus.cc:504] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.144625 4033 leader_election.cc:288] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.144901 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.144917 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.144999 3940 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.145005 3790 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.145228 3829 leader_election.cc:302] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.145375 4033 raft_consensus.cc:2781] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.145428 4033 raft_consensus.cc:482] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.145473 4033 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.146145 4033 raft_consensus.cc:504] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.146414 4033 leader_election.cc:288] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.146699 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.146705 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.146785 3940 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.146811 3790 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.147416 3790 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.147406 3940 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.147676 3831 leader_election.cc:302] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.147821 4033 raft_consensus.cc:2781] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.147923 4033 raft_consensus.cc:686] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:08.148038 4033 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.149395 3685 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.157349 4033 raft_consensus.cc:482] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.157438 4033 raft_consensus.cc:504] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.157740 4033 leader_election.cc:288] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.158010 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.158015 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1" is_pre_election: true
I20240501 08:00:08.158115 3790 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.158120 3940 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 0.
I20240501 08:00:08.158344 3831 leader_election.cc:302] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.158488 4033 raft_consensus.cc:2781] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.158541 4033 raft_consensus.cc:482] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.158577 4033 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.159236 4033 raft_consensus.cc:504] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.159510 4033 leader_election.cc:288] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:08.159757 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.159776 3940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.159848 3790 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.159878 3940 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.160441 3790 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.160513 3940 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 1.
I20240501 08:00:08.160652 3831 leader_election.cc:302] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.160801 4033 raft_consensus.cc:2781] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.160882 4033 raft_consensus.cc:686] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:08.161000 4033 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.162326 3685 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.309175 4041 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:00:08.309307 4041 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:08.309662 4041 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:08.309974 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97" is_pre_election: true
I20240501 08:00:08.310021 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f" is_pre_election: true
I20240501 08:00:08.310087 3866 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.310127 3790 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 0.
I20240501 08:00:08.310317 3904 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.310467 4041 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:00:08.310523 4041 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:00:08.310555 4041 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.311264 4041 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:08.311542 4041 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:08.311805 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:08.311805 3866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.311931 3790 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.311935 3866 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:00:08.312726 3790 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.312722 3866 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 1.
I20240501 08:00:08.313014 3906 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:08.313161 4041 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:00:08.313238 4041 raft_consensus.cc:686] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.313386 4041 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:08.314841 3685 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.423539 3790 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.424412 4043 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.424546 3789 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.425093 3864 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.425395 3790 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.425688 3864 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.425966 3866 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.426115 3993 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.426371 3993 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.426605 3993 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.426669 3866 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.426981 3865 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.427371 3788 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.427636 3787 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.427781 3789 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.428077 3864 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.428223 3993 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.428452 3993 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.429826 4043 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.430054 4043 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.430276 4043 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.430968 4041 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.431195 4041 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.431412 4041 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.437353 3940 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.437635 3941 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.438287 3860 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.438568 3859 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.439803 4030 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.440052 4030 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.440289 4030 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.440510 4030 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.444690 3939 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.445206 3788 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.445714 3939 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.446144 4033 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.447481 3939 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.447975 4031 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.449999 3938 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.450559 3990 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.450855 3990 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.453882 3990 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.454097 3782 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.453919 3783 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.454324 3788 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:00:08.454811 3990 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.455096 3990 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.455219 4069 mvcc.cc:205] Tried to move back new op lower bound from 7022798472986079232 to 7022798471278252032. Current Snapshot: MvccSnapshot[applied={T|T < 7022798472986079232}]
I20240501 08:00:08.455317 3990 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:00:08.470005 4079 mvcc.cc:205] Tried to move back new op lower bound from 7022798472960745472 to 7022798471671296000. Current Snapshot: MvccSnapshot[applied={T|T < 7022798472949194752}]
I20240501 08:00:08.483368 4067 mvcc.cc:205] Tried to move back new op lower bound from 7022798472917061632 to 7022798471731539968. Current Snapshot: MvccSnapshot[applied={T|T < 7022798472910372864}]
I20240501 08:00:08.770076 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:08.778874 3941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47826
I20240501 08:00:08.779026 3941 raft_consensus.cc:595] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
W20240501 08:00:08.780115 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.781708 3941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47826
I20240501 08:00:08.781826 3941 raft_consensus.cc:595] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
I20240501 08:00:08.782056 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 08:00:08.782184 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 08:00:08.784571 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.800869 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.802156 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.802713 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.818281 3912 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.818320 3917 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.820523 3920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.821009 4072 raft_consensus.cc:979] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:08.821086 4072 raft_consensus.cc:979] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:08.821130 4041 raft_consensus.cc:1067] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:08.821131 4072 raft_consensus.cc:1067] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
W20240501 08:00:08.826277 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.826825 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.827333 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.827837 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.829047 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.829071 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.829927 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.832736 3788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34758
I20240501 08:00:08.832895 3788 raft_consensus.cc:482] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:08.832964 3788 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:00:08.833446 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.834147 3788 raft_consensus.cc:504] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
W20240501 08:00:08.834204 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.834183 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.834487 3788 leader_election.cc:288] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
W20240501 08:00:08.834705 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.834887 3788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34758
I20240501 08:00:08.834983 3788 raft_consensus.cc:482] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:08.835043 3788 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:00:08.835497 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.835862 3788 raft_consensus.cc:504] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.836169 3788 leader_election.cc:288] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
W20240501 08:00:08.836656 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
W20240501 08:00:08.837164 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.840162 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.840405 3941 raft_consensus.cc:3032] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:08.840462 3941 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.840580 3941 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.840719 3941 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:00:08.841463 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Service unavailable: leader transfer in progress
I20240501 08:00:08.841768 3941 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:08.842307 3941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:08.842404 3941 raft_consensus.cc:3032] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:08.842450 3941 raft_consensus.cc:729] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:08.842594 3941 consensus_queue.cc:260] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 21, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.842768 3941 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:00:08.843132 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.843794 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.842779 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
I20240501 08:00:08.843859 3941 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
W20240501 08:00:08.845000 3913 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.845000 3921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.845470 3920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.851631 3920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47812: Illegal state: replica f818705fd267423dba3ae9ae51d3a8a1 is not leader of this config: current role FOLLOWER
I20240501 08:00:08.853742 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c853e31bc74cb38778ed1ab133d966" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.853895 3862 raft_consensus.cc:3037] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:08.853766 3859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:08.854065 3859 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:08.854768 3862 raft_consensus.cc:2454] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:08.855041 3859 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
W20240501 08:00:08.855939 3765 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
W20240501 08:00:08.856626 3765 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
W20240501 08:00:08.856748 3766 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
W20240501 08:00:08.858971 3765 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
W20240501 08:00:08.859055 3766 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
W20240501 08:00:08.859477 3765 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34772: Illegal state: replica 504c682c767543748142f114ffa7be2f is not leader of this config: current role FOLLOWER
I20240501 08:00:08.861584 3753 leader_election.cc:302] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.861745 4100 raft_consensus.cc:2781] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:08.861817 4100 raft_consensus.cc:686] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:08.861939 4100 consensus_queue.cc:237] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 21, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.862560 3753 leader_election.cc:302] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:08.862715 4000 raft_consensus.cc:2781] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:08.862782 4000 raft_consensus.cc:686] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:08.862897 4000 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:08.863483 3684 catalog_manager.cc:5484] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 1 to 2, leader changed from f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.868245 3937 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
W20240501 08:00:08.868783 3842 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.868837 3844 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.868854 3840 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
I20240501 08:00:08.870465 4099 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20240501 08:00:08.870471 3864 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
W20240501 08:00:08.870903 3840 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
W20240501 08:00:08.871659 3840 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
I20240501 08:00:08.872092 4100 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20240501 08:00:08.874017 3840 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59554: Illegal state: replica d5c9f893d7344b4f841e68668e19ab97 is not leader of this config: current role FOLLOWER
I20240501 08:00:08.875550 4065 mvcc.cc:205] Tried to move back new op lower bound from 7022798474717904896 to 7022798474700034048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798474717904896 or (T in {7022798474717904896})}]
I20240501 08:00:08.878428 3678 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 1 to 2, leader changed from f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:08.884102 3862 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240501 08:00:08.884377 3941 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240501 08:00:08.885262 4099 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 08:00:08.886229 4100 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 08:00:08.946507 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:08.948427 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:08.949038 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:08.949565 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:08.949759 396 server_base.cc:1034] running on GCE node
I20240501 08:00:08.949898 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:08.949934 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:08.949947 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550408949947 us; error 0 us; skew 500 ppm
I20240501 08:00:08.950904 396 webserver.cc:460] Webserver started at http://127.0.99.4:45519/ using document root <none> and password file <none>
I20240501 08:00:08.951041 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:08.951093 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:08.951164 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:08.951512 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-3-root/instance:
uuid: "e34080744cfb46bc8cb6c41de486c343"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:08.952821 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:08.953619 4149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.953845 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.953907 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-3-root
uuid: "e34080744cfb46bc8cb6c41de486c343"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:08.953970 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:08.968881 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:08.969274 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:08.969552 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:08.969995 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:08.970034 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.970077 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:08.970105 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.974457 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:43005
I20240501 08:00:08.974498 4212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:43005 every 8 connection(s)
I20240501 08:00:08.975477 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:08.977180 4218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:08.977726 4217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:08.978212 4220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:08.978714 396 server_base.cc:1034] running on GCE node
I20240501 08:00:08.978883 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:08.978920 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:08.978940 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550408978940 us; error 0 us; skew 500 ppm
I20240501 08:00:08.979604 4213 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:08.979692 4213 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:08.979905 4213 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:08.980016 396 webserver.cc:460] Webserver started at http://127.0.99.5:36945/ using document root <none> and password file <none>
I20240501 08:00:08.980165 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:08.980219 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:08.980293 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:08.980545 3678 ts_manager.cc:194] Registered new tserver with Master: e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005)
I20240501 08:00:08.980685 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-4-root/instance:
uuid: "385e9e7fb97848e69d579dd53df27aae"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:08.981212 3678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41218
I20240501 08:00:08.982161 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:08.982936 4225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.983144 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:08.983201 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-4-root
uuid: "385e9e7fb97848e69d579dd53df27aae"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:08.983263 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:08.993647 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:08.994022 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:08.994280 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:08.994743 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:08.994782 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.994815 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:08.994843 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:08.999608 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.5:42185
I20240501 08:00:08.999692 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.5:42185 every 8 connection(s)
I20240501 08:00:09.000393 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:00:09.001936 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:09.002380 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:00:09.003229 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:00:09.003854 396 server_base.cc:1034] running on GCE node
I20240501 08:00:09.004010 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:00:09.004047 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:00:09.004065 396 hybrid_clock.cc:648] HybridClock initialized: now 1714550409004065 us; error 0 us; skew 500 ppm
I20240501 08:00:09.005520 396 webserver.cc:460] Webserver started at http://127.0.99.6:37637/ using document root <none> and password file <none>
I20240501 08:00:09.005669 396 fs_manager.cc:359] Metadata directory not provided
I20240501 08:00:09.005724 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:00:09.005800 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:00:09.005911 4289 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:09.006001 4289 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:09.006193 4289 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:09.006202 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-5-root/instance:
uuid: "21f956c9a93b4bf19575304199b36492"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:09.006793 3678 ts_manager.cc:194] Registered new tserver with Master: 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185)
I20240501 08:00:09.007283 3678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41224
I20240501 08:00:09.007737 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:09.008579 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:09.008793 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:09.008857 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-5-root
uuid: "21f956c9a93b4bf19575304199b36492"
format_stamp: "Formatted at 2024-05-01 08:00:08 on dist-test-slave-rp4l"
I20240501 08:00:09.008929 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:00:09.032366 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:00:09.032783 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:00:09.033039 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:00:09.033502 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:00:09.033545 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:09.033586 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:00:09.033613 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:00:09.038745 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.6:43819
I20240501 08:00:09.038841 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.6:43819 every 8 connection(s)
I20240501 08:00:09.042932 4364 heartbeater.cc:347] Connected to a master server at 127.0.99.62:34785
I20240501 08:00:09.043026 4364 heartbeater.cc:464] Registering TS with master...
I20240501 08:00:09.043228 4364 heartbeater.cc:511] Master 127.0.99.62:34785 requested a full tablet report, sending...
I20240501 08:00:09.043803 3678 ts_manager.cc:194] Registered new tserver with Master: 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819)
I20240501 08:00:09.044289 3678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41230
I20240501 08:00:09.783115 3862 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:09.784443 3937 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:00:09.784636 3790 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 08:00:09.784780 4369 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:00:09.784948 4139 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 08:00:09.786396 4139 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.786757 3937 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.786880 3790 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.789001 3678 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index -1 to 27, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
W20240501 08:00:09.790494 3830 consensus_peers.cc:485] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 728c3dff60a8498e99bc434ff175083f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.792819 3937 consensus_queue.cc:237] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:09.793968 3790 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.794303 4377 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.794528 3862 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.794970 4377 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.795934 3790 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.796674 3862 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.795603 4378 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.797911 3680 catalog_manager.cc:5484] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
W20240501 08:00:09.798965 3904 consensus_peers.cc:485] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d827adaf117470fa9fada0d8f245671. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.799508 3937 consensus_queue.cc:237] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:09.800891 3790 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.801044 3862 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.801366 4130 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.801693 4130 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.803118 4130 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:09.803496 3790 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
W20240501 08:00:09.805110 3906 consensus_peers.cc:485] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1ef543279a8493295c4439e6d8d87cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.805370 3684 catalog_manager.cc:5484] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index -1 to 29, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.805652 3862 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:09.809864 3790 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:09.811095 3937 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:09.811136 3862 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:09.811417 4384 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:09.811663 4383 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:09.812963 4000 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:09.814479 3937 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
W20240501 08:00:09.814592 3755 consensus_peers.cc:485] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc118979132748e1a45e3e33e655ac11. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.814811 3862 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:09.815018 3678 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:09.816509 3790 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:09.817649 3862 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.817693 3937 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.818037 4383 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.818363 4383 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.819608 4383 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.819928 3937 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.820308 3864 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:09.822010 3680 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
W20240501 08:00:09.822913 3754 consensus_peers.cc:485] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 824019b3e8674c329bdeb8467db1f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.823513 3862 consensus_queue.cc:237] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:09.825016 3790 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.825254 3937 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:09.825294 4370 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.825592 4370 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:09.826484 4139 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.827365 3937 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.828505 3684 catalog_manager.cc:5484] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:09.828867 3829 consensus_peers.cc:485] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b909fdab57ca42a0b05c41dde754743a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.829344 3790 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:09.981984 4213 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:10.008059 4289 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:10.044987 4364 heartbeater.cc:503] Master 127.0.99.62:34785 was elected leader, sending a full tablet report...
I20240501 08:00:10.184357 4386 ts_tablet_manager.cc:902] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Initiating tablet copy from peer f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:10.184859 4386 tablet_copy_client.cc:308] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:39165
I20240501 08:00:10.188540 3951 tablet_copy_service.cc:140] P f818705fd267423dba3ae9ae51d3a8a1: Received BeginTabletCopySession request for tablet 9d827adaf117470fa9fada0d8f245671 from peer 385e9e7fb97848e69d579dd53df27aae ({username='slave'} at 127.0.0.1:35776)
I20240501 08:00:10.188644 3951 tablet_copy_service.cc:161] P f818705fd267423dba3ae9ae51d3a8a1: Beginning new tablet copy session on tablet 9d827adaf117470fa9fada0d8f245671 from peer 385e9e7fb97848e69d579dd53df27aae at {username='slave'} at 127.0.0.1:35776: session id = 385e9e7fb97848e69d579dd53df27aae-9d827adaf117470fa9fada0d8f245671
I20240501 08:00:10.189671 3951 tablet_copy_source_session.cc:205] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.190373 4386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d827adaf117470fa9fada0d8f245671. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.192591 4386 tablet_copy_client.cc:783] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.192749 4386 tablet_copy_client.cc:647] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.193619 4390 tablet_copy_client.cc:1002] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.231331 4392 ts_tablet_manager.cc:902] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Initiating tablet copy from peer 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:10.231861 4392 tablet_copy_client.cc:308] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39565
I20240501 08:00:10.235703 3800 tablet_copy_service.cc:140] P 504c682c767543748142f114ffa7be2f: Received BeginTabletCopySession request for tablet 824019b3e8674c329bdeb8467db1f6ff from peer 21f956c9a93b4bf19575304199b36492 ({username='slave'} at 127.0.0.1:52002)
I20240501 08:00:10.235811 3800 tablet_copy_service.cc:161] P 504c682c767543748142f114ffa7be2f: Beginning new tablet copy session on tablet 824019b3e8674c329bdeb8467db1f6ff from peer 21f956c9a93b4bf19575304199b36492 at {username='slave'} at 127.0.0.1:52002: session id = 21f956c9a93b4bf19575304199b36492-824019b3e8674c329bdeb8467db1f6ff
I20240501 08:00:10.236824 3800 tablet_copy_source_session.cc:205] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.237613 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824019b3e8674c329bdeb8467db1f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.239746 4392 tablet_copy_client.cc:783] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.239897 4392 tablet_copy_client.cc:647] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.240773 4395 tablet_copy_client.cc:1002] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.388860 4398 ts_tablet_manager.cc:902] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Initiating tablet copy from peer f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:10.389397 4398 tablet_copy_client.cc:308] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:39165
I20240501 08:00:10.392230 4404 ts_tablet_manager.cc:902] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:10.393208 4404 tablet_copy_client.cc:308] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34681
I20240501 08:00:10.393226 3951 tablet_copy_service.cc:140] P f818705fd267423dba3ae9ae51d3a8a1: Received BeginTabletCopySession request for tablet f1ef543279a8493295c4439e6d8d87cd from peer e34080744cfb46bc8cb6c41de486c343 ({username='slave'} at 127.0.0.1:35780)
I20240501 08:00:10.393435 3951 tablet_copy_service.cc:161] P f818705fd267423dba3ae9ae51d3a8a1: Beginning new tablet copy session on tablet f1ef543279a8493295c4439e6d8d87cd from peer e34080744cfb46bc8cb6c41de486c343 at {username='slave'} at 127.0.0.1:35780: session id = e34080744cfb46bc8cb6c41de486c343-f1ef543279a8493295c4439e6d8d87cd
I20240501 08:00:10.394466 3951 tablet_copy_source_session.cc:205] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.395287 4398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ef543279a8493295c4439e6d8d87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.397508 4398 tablet_copy_client.cc:783] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.397650 4398 tablet_copy_client.cc:647] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.398490 4400 tablet_copy_client.cc:1002] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.399194 4410 ts_tablet_manager.cc:902] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:10.399250 3876 tablet_copy_service.cc:140] P d5c9f893d7344b4f841e68668e19ab97: Received BeginTabletCopySession request for tablet b909fdab57ca42a0b05c41dde754743a from peer 385e9e7fb97848e69d579dd53df27aae ({username='slave'} at 127.0.0.1:47650)
I20240501 08:00:10.399338 3876 tablet_copy_service.cc:161] P d5c9f893d7344b4f841e68668e19ab97: Beginning new tablet copy session on tablet b909fdab57ca42a0b05c41dde754743a from peer 385e9e7fb97848e69d579dd53df27aae at {username='slave'} at 127.0.0.1:47650: session id = 385e9e7fb97848e69d579dd53df27aae-b909fdab57ca42a0b05c41dde754743a
I20240501 08:00:10.400043 4410 tablet_copy_client.cc:308] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34681
I20240501 08:00:10.400121 3876 tablet_copy_source_session.cc:205] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.401324 4404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909fdab57ca42a0b05c41dde754743a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.403246 4404 tablet_copy_client.cc:783] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.403375 4404 tablet_copy_client.cc:647] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.404050 3876 tablet_copy_service.cc:140] P d5c9f893d7344b4f841e68668e19ab97: Received BeginTabletCopySession request for tablet 728c3dff60a8498e99bc434ff175083f from peer 21f956c9a93b4bf19575304199b36492 ({username='slave'} at 127.0.0.1:47656)
I20240501 08:00:10.404142 3876 tablet_copy_service.cc:161] P d5c9f893d7344b4f841e68668e19ab97: Beginning new tablet copy session on tablet 728c3dff60a8498e99bc434ff175083f from peer 21f956c9a93b4bf19575304199b36492 at {username='slave'} at 127.0.0.1:47656: session id = 21f956c9a93b4bf19575304199b36492-728c3dff60a8498e99bc434ff175083f
I20240501 08:00:10.404771 4407 tablet_copy_client.cc:1002] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.404937 3876 tablet_copy_source_session.cc:205] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.406311 4410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728c3dff60a8498e99bc434ff175083f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.408113 4410 tablet_copy_client.cc:783] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.408247 4410 tablet_copy_client.cc:647] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.409348 4411 tablet_copy_client.cc:1002] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.410775 4415 ts_tablet_manager.cc:902] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Initiating tablet copy from peer 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:10.411375 4415 tablet_copy_client.cc:308] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39565
I20240501 08:00:10.415136 3800 tablet_copy_service.cc:140] P 504c682c767543748142f114ffa7be2f: Received BeginTabletCopySession request for tablet dc118979132748e1a45e3e33e655ac11 from peer e34080744cfb46bc8cb6c41de486c343 ({username='slave'} at 127.0.0.1:52014)
I20240501 08:00:10.415226 3800 tablet_copy_service.cc:161] P 504c682c767543748142f114ffa7be2f: Beginning new tablet copy session on tablet dc118979132748e1a45e3e33e655ac11 from peer e34080744cfb46bc8cb6c41de486c343 at {username='slave'} at 127.0.0.1:52014: session id = e34080744cfb46bc8cb6c41de486c343-dc118979132748e1a45e3e33e655ac11
I20240501 08:00:10.416039 3800 tablet_copy_source_session.cc:205] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.416747 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc118979132748e1a45e3e33e655ac11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.418852 4415 tablet_copy_client.cc:783] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.419004 4415 tablet_copy_client.cc:647] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.419894 4419 tablet_copy_client.cc:1002] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:10.782420 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:10.784011 3937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
mode: GRACEFUL
new_leader_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:47826
I20240501 08:00:10.784230 3937 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Received request to transfer leadership to TS d5c9f893d7344b4f841e68668e19ab97
I20240501 08:00:10.784780 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 08:00:10.784881 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:11.078732 4420 raft_consensus.cc:979] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1: : Instructing follower d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:11.078868 4421 raft_consensus.cc:1067] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Signalling peer d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:11.079237 3862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:59544
I20240501 08:00:11.079353 3862 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:11.079416 3862 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:11.080204 3862 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:11.080519 3862 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 2 election: Requested vote from peers f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165), 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:11.080829 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:11.080915 3788 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:11.081606 3788 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 2.
I20240501 08:00:11.081956 3831 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:11.082072 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:11.082155 4423 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:11.082175 3937 raft_consensus.cc:3032] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:11.082248 3937 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:11.082263 4423 raft_consensus.cc:686] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:11.082377 3937 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:11.082559 3937 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:11.082393 4423 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } }
I20240501 08:00:11.083593 3937 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 2.
I20240501 08:00:11.084281 3678 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 1 to 2, leader changed from f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:11.652076 3937 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:00:11.652420 4139 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:11.654974 3788 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:00:11.655305 4139 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:12.195139 4386 tablet_copy_client.cc:523] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.196367 4386 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap starting.
I20240501 08:00:12.220276 4386 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=123 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.220552 4386 tablet_bootstrap.cc:492] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap complete.
I20240501 08:00:12.220757 4386 ts_tablet_manager.cc:1367] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 08:00:12.221171 4386 raft_consensus.cc:348] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:12.221307 4386 raft_consensus.cc:729] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Initialized, Role: LEARNER
I20240501 08:00:12.221428 4386 consensus_queue.cc:260] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:12.221753 4386 ts_tablet_manager.cc:1398] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:00:12.222151 3951 tablet_copy_service.cc:342] P f818705fd267423dba3ae9ae51d3a8a1: Request end of tablet copy session 385e9e7fb97848e69d579dd53df27aae-9d827adaf117470fa9fada0d8f245671 received from {username='slave'} at 127.0.0.1:35776
I20240501 08:00:12.222234 3951 tablet_copy_service.cc:433] P f818705fd267423dba3ae9ae51d3a8a1: ending tablet copy session 385e9e7fb97848e69d579dd53df27aae-9d827adaf117470fa9fada0d8f245671 on tablet 9d827adaf117470fa9fada0d8f245671 with peer 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:12.241633 4392 tablet_copy_client.cc:523] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.242934 4392 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Bootstrap starting.
I20240501 08:00:12.265204 4392 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.265513 4392 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Bootstrap complete.
I20240501 08:00:12.265707 4392 ts_tablet_manager.cc:1367] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.003s
I20240501 08:00:12.266136 4392 raft_consensus.cc:348] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:12.266263 4392 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Initialized, Role: LEARNER
I20240501 08:00:12.266360 4392 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:12.266692 4392 ts_tablet_manager.cc:1398] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 08:00:12.267102 3800 tablet_copy_service.cc:342] P 504c682c767543748142f114ffa7be2f: Request end of tablet copy session 21f956c9a93b4bf19575304199b36492-824019b3e8674c329bdeb8467db1f6ff received from {username='slave'} at 127.0.0.1:52002
I20240501 08:00:12.267169 3800 tablet_copy_service.cc:433] P 504c682c767543748142f114ffa7be2f: ending tablet copy session 21f956c9a93b4bf19575304199b36492-824019b3e8674c329bdeb8467db1f6ff on tablet 824019b3e8674c329bdeb8467db1f6ff with peer 21f956c9a93b4bf19575304199b36492
I20240501 08:00:12.310349 4338 raft_consensus.cc:1203] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:00:12.398773 4398 tablet_copy_client.cc:523] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.400355 4398 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Bootstrap starting.
I20240501 08:00:12.405092 4404 tablet_copy_client.cc:523] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.406369 4404 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Bootstrap starting.
I20240501 08:00:12.409586 4410 tablet_copy_client.cc:523] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.411034 4410 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Bootstrap starting.
I20240501 08:00:12.420125 4415 tablet_copy_client.cc:523] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.421442 4415 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Bootstrap starting.
I20240501 08:00:12.424961 4398 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.425196 4398 tablet_bootstrap.cc:492] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Bootstrap complete.
I20240501 08:00:12.425554 4398 ts_tablet_manager.cc:1367] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20240501 08:00:12.426002 4398 raft_consensus.cc:348] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:12.426119 4398 raft_consensus.cc:729] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e34080744cfb46bc8cb6c41de486c343, State: Initialized, Role: LEARNER
I20240501 08:00:12.426263 4398 consensus_queue.cc:260] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:12.426725 4398 ts_tablet_manager.cc:1398] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:12.428391 3951 tablet_copy_service.cc:342] P f818705fd267423dba3ae9ae51d3a8a1: Request end of tablet copy session e34080744cfb46bc8cb6c41de486c343-f1ef543279a8493295c4439e6d8d87cd received from {username='slave'} at 127.0.0.1:35780
I20240501 08:00:12.428463 3951 tablet_copy_service.cc:433] P f818705fd267423dba3ae9ae51d3a8a1: ending tablet copy session e34080744cfb46bc8cb6c41de486c343-f1ef543279a8493295c4439e6d8d87cd on tablet f1ef543279a8493295c4439e6d8d87cd with peer e34080744cfb46bc8cb6c41de486c343
I20240501 08:00:12.436563 4410 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.436813 4404 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.436844 4410 tablet_bootstrap.cc:492] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Bootstrap complete.
I20240501 08:00:12.437083 4404 tablet_bootstrap.cc:492] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Bootstrap complete.
I20240501 08:00:12.437186 4410 ts_tablet_manager.cc:1367] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.003s
I20240501 08:00:12.437247 4404 ts_tablet_manager.cc:1367] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.004s
I20240501 08:00:12.437701 4404 raft_consensus.cc:348] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:12.437701 4410 raft_consensus.cc:348] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:12.437871 4404 raft_consensus.cc:729] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Initialized, Role: LEARNER
I20240501 08:00:12.437922 4410 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Initialized, Role: LEARNER
I20240501 08:00:12.437995 4404 consensus_queue.cc:260] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:12.438023 4410 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:12.438414 4410 ts_tablet_manager.cc:1398] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.438686 4404 ts_tablet_manager.cc:1398] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.438817 3876 tablet_copy_service.cc:342] P d5c9f893d7344b4f841e68668e19ab97: Request end of tablet copy session 21f956c9a93b4bf19575304199b36492-728c3dff60a8498e99bc434ff175083f received from {username='slave'} at 127.0.0.1:47656
I20240501 08:00:12.439160 3876 tablet_copy_service.cc:433] P d5c9f893d7344b4f841e68668e19ab97: ending tablet copy session 21f956c9a93b4bf19575304199b36492-728c3dff60a8498e99bc434ff175083f on tablet 728c3dff60a8498e99bc434ff175083f with peer 21f956c9a93b4bf19575304199b36492
I20240501 08:00:12.439999 3876 tablet_copy_service.cc:342] P d5c9f893d7344b4f841e68668e19ab97: Request end of tablet copy session 385e9e7fb97848e69d579dd53df27aae-b909fdab57ca42a0b05c41dde754743a received from {username='slave'} at 127.0.0.1:47650
I20240501 08:00:12.440064 3876 tablet_copy_service.cc:433] P d5c9f893d7344b4f841e68668e19ab97: ending tablet copy session 385e9e7fb97848e69d579dd53df27aae-b909fdab57ca42a0b05c41dde754743a on tablet b909fdab57ca42a0b05c41dde754743a with peer 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:12.455724 4415 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.456035 4415 tablet_bootstrap.cc:492] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Bootstrap complete.
I20240501 08:00:12.456262 4415 ts_tablet_manager.cc:1367] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.005s
I20240501 08:00:12.456681 4415 raft_consensus.cc:348] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:12.456789 4415 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e34080744cfb46bc8cb6c41de486c343, State: Initialized, Role: LEARNER
I20240501 08:00:12.456876 4415 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:12.457114 4415 ts_tablet_manager.cc:1398] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:12.457568 3800 tablet_copy_service.cc:342] P 504c682c767543748142f114ffa7be2f: Request end of tablet copy session e34080744cfb46bc8cb6c41de486c343-dc118979132748e1a45e3e33e655ac11 received from {username='slave'} at 127.0.0.1:52014
I20240501 08:00:12.457645 3800 tablet_copy_service.cc:433] P 504c682c767543748142f114ffa7be2f: ending tablet copy session e34080744cfb46bc8cb6c41de486c343-dc118979132748e1a45e3e33e655ac11 on tablet dc118979132748e1a45e3e33e655ac11 with peer e34080744cfb46bc8cb6c41de486c343
I20240501 08:00:12.504693 4187 raft_consensus.cc:1203] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 08:00:12.531545 4187 raft_consensus.cc:1203] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:00:12.638837 4263 raft_consensus.cc:1203] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:00:12.785113 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:12.786299 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 08:00:12.786406 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:12.879441 4422 raft_consensus.cc:1050] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: attempting to promote NON_VOTER 21f956c9a93b4bf19575304199b36492 to VOTER
I20240501 08:00:12.879822 4422 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:12.881096 4263 raft_consensus.cc:1203] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:00:12.881253 4338 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:12.881253 3864 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:12.881628 4446 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:12.882009 3939 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:12.882279 4422 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:12.883477 4446 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:12.884411 4448 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.884824 3864 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.884922 4338 raft_consensus.cc:1203] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 08:00:12.886240 4338 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.887854 3939 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.888125 3684 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 29 to 30, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.894449 3790 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:12.895578 4338 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:12.895855 3939 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:12.895910 4446 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:12.896201 4449 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:12.897420 4448 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.897696 4338 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.897913 3939 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.899420 3673 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 824019b3e8674c329bdeb8467db1f6ff with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:12.899745 3687 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 reported cstate change: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:12.901937 3856 tablet_service.cc:1514] Processing DeleteTablet for tablet 824019b3e8674c329bdeb8467db1f6ff with delete_type TABLET_DATA_TOMBSTONED (TS d5c9f893d7344b4f841e68668e19ab97 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:12.902329 4458 tablet_replica.cc:330] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: stopping tablet replica
I20240501 08:00:12.902468 4458 raft_consensus.cc:2229] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:12.902765 4458 raft_consensus.cc:2258] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:12.903452 4458 ts_tablet_manager.cc:1875] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:12.905213 4458 ts_tablet_manager.cc:1888] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 08:00:12.905287 4458 log.cc:1196] T 824019b3e8674c329bdeb8467db1f6ff P d5c9f893d7344b4f841e68668e19ab97: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/wals/824019b3e8674c329bdeb8467db1f6ff
I20240501 08:00:12.905704 3671 catalog_manager.cc:4830] TS d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681): tablet 824019b3e8674c329bdeb8467db1f6ff (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:13.004762 4446 raft_consensus.cc:1050] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: attempting to promote NON_VOTER e34080744cfb46bc8cb6c41de486c343 to VOTER
I20240501 08:00:13.005208 4446 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 22, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:13.006522 3939 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:13.006590 4187 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:13.006639 3864 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:13.006986 4457 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.007189 4449 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.007354 4446 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.009236 4448 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.010505 4187 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.011118 3864 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.012560 3680 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 reported cstate change: config changed from index 30 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.012856 3939 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.016992 3790 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 22, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:13.017995 4187 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:13.018119 3864 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:13.018302 4457 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:13.018752 4457 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:13.020215 4457 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.020488 4187 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.021246 3862 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.021641 3673 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc118979132748e1a45e3e33e655ac11 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:13.022912 3687 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 reported cstate change: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.024770 3927 tablet_service.cc:1514] Processing DeleteTablet for tablet dc118979132748e1a45e3e33e655ac11 with delete_type TABLET_DATA_TOMBSTONED (TS f818705fd267423dba3ae9ae51d3a8a1 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:47780
I20240501 08:00:13.025211 4463 tablet_replica.cc:330] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: stopping tablet replica
I20240501 08:00:13.025336 4463 raft_consensus.cc:2229] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:13.025501 4463 raft_consensus.cc:2258] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:13.026008 4463 ts_tablet_manager.cc:1875] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:13.027731 4463 ts_tablet_manager.cc:1888] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 08:00:13.027796 4463 log.cc:1196] T dc118979132748e1a45e3e33e655ac11 P f818705fd267423dba3ae9ae51d3a8a1: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/wals/dc118979132748e1a45e3e33e655ac11
I20240501 08:00:13.028177 3671 catalog_manager.cc:4830] TS f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165): tablet dc118979132748e1a45e3e33e655ac11 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:13.095304 4465 raft_consensus.cc:1050] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1: attempting to promote NON_VOTER e34080744cfb46bc8cb6c41de486c343 to VOTER
I20240501 08:00:13.095800 4465 consensus_queue.cc:237] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:13.097010 3862 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.097046 4187 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.097345 4464 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.097430 3790 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.097543 4445 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.097827 4445 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.099479 4464 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.099776 3790 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.101024 4187 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.101667 3862 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.101886 3678 catalog_manager.cc:5484] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index 29 to 30, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:13.106812 3939 consensus_queue.cc:237] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:13.107708 3862 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.107757 4187 raft_consensus.cc:1261] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.108039 4465 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.108178 4445 consensus_queue.cc:1035] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.109604 4465 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.109776 3862 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.111163 4187 raft_consensus.cc:2932] T f1ef543279a8493295c4439e6d8d87cd P e34080744cfb46bc8cb6c41de486c343 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:13.111527 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1ef543279a8493295c4439e6d8d87cd with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:13.111714 3687 catalog_manager.cc:5484] T f1ef543279a8493295c4439e6d8d87cd P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:13.113930 3777 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ef543279a8493295c4439e6d8d87cd with delete_type TABLET_DATA_TOMBSTONED (TS 504c682c767543748142f114ffa7be2f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:13.114338 4470 tablet_replica.cc:330] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: stopping tablet replica
I20240501 08:00:13.114477 4470 raft_consensus.cc:2229] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:13.114640 4470 raft_consensus.cc:2258] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:13.115214 4470 ts_tablet_manager.cc:1875] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:13.117225 4470 ts_tablet_manager.cc:1888] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 08:00:13.117300 4470 log.cc:1196] T f1ef543279a8493295c4439e6d8d87cd P 504c682c767543748142f114ffa7be2f: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/wals/f1ef543279a8493295c4439e6d8d87cd
I20240501 08:00:13.117761 3673 catalog_manager.cc:4830] TS 504c682c767543748142f114ffa7be2f (127.0.99.1:39565): tablet f1ef543279a8493295c4439e6d8d87cd (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:13.131701 4465 raft_consensus.cc:1050] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1: attempting to promote NON_VOTER 385e9e7fb97848e69d579dd53df27aae to VOTER
I20240501 08:00:13.132109 4465 consensus_queue.cc:237] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:13.133463 4263 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.133517 3790 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.133487 3862 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.133886 4464 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.134068 4445 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.134207 4465 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.135965 4465 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.136253 3790 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.136355 3864 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.137887 4263 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.138087 3680 catalog_manager.cc:5484] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.141043 3939 consensus_queue.cc:237] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:13.142104 3790 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.142171 4263 raft_consensus.cc:1261] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.142479 4464 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.143093 4464 consensus_queue.cc:1035] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.143868 4472 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.144253 4263 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.144919 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d827adaf117470fa9fada0d8f245671 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:13.145852 3680 catalog_manager.cc:5484] T 9d827adaf117470fa9fada0d8f245671 P 385e9e7fb97848e69d579dd53df27aae reported cstate change: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.146476 3790 raft_consensus.cc:2932] T 9d827adaf117470fa9fada0d8f245671 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.147851 3856 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d827adaf117470fa9fada0d8f245671 with delete_type TABLET_DATA_TOMBSTONED (TS d5c9f893d7344b4f841e68668e19ab97 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:13.148047 4458 tablet_replica.cc:330] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: stopping tablet replica
I20240501 08:00:13.148180 4458 raft_consensus.cc:2229] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:13.148319 4458 raft_consensus.cc:2258] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:13.149250 4458 ts_tablet_manager.cc:1875] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:13.151244 4458 ts_tablet_manager.cc:1888] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 08:00:13.151331 4458 log.cc:1196] T 9d827adaf117470fa9fada0d8f245671 P d5c9f893d7344b4f841e68668e19ab97: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/wals/9d827adaf117470fa9fada0d8f245671
I20240501 08:00:13.151753 3671 catalog_manager.cc:4830] TS d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681): tablet 9d827adaf117470fa9fada0d8f245671 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:13.257678 4476 raft_consensus.cc:1050] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: attempting to promote NON_VOTER 385e9e7fb97848e69d579dd53df27aae to VOTER
I20240501 08:00:13.258108 4476 consensus_queue.cc:237] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:13.259322 3790 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.259346 3939 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.259349 4263 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 08:00:13.259723 4479 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.259948 4477 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.260178 4436 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:13.261758 4479 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.262164 4263 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.262533 3790 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.263967 3687 catalog_manager.cc:5484] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae reported cstate change: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.264698 3939 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.267313 3864 consensus_queue.cc:237] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:13.268350 3939 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.268352 4263 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 08:00:13.268909 4436 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.269308 4476 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:13.270450 4436 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.270849 3939 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.272135 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b909fdab57ca42a0b05c41dde754743a with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:13.272756 4263 raft_consensus.cc:2932] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:13.272786 3680 catalog_manager.cc:5484] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 30 to 31, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:13.275043 3777 tablet_service.cc:1514] Processing DeleteTablet for tablet b909fdab57ca42a0b05c41dde754743a with delete_type TABLET_DATA_TOMBSTONED (TS 504c682c767543748142f114ffa7be2f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:13.275256 4470 tablet_replica.cc:330] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: stopping tablet replica
I20240501 08:00:13.275398 4470 raft_consensus.cc:2229] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:13.275525 4470 raft_consensus.cc:2258] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:13.276121 4470 ts_tablet_manager.cc:1875] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:13.278101 4470 ts_tablet_manager.cc:1888] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 08:00:13.278177 4470 log.cc:1196] T b909fdab57ca42a0b05c41dde754743a P 504c682c767543748142f114ffa7be2f: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/wals/b909fdab57ca42a0b05c41dde754743a
I20240501 08:00:13.278650 3673 catalog_manager.cc:4830] TS 504c682c767543748142f114ffa7be2f (127.0.99.1:39565): tablet b909fdab57ca42a0b05c41dde754743a (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:13.342504 4477 raft_consensus.cc:1050] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: attempting to promote NON_VOTER 21f956c9a93b4bf19575304199b36492 to VOTER
I20240501 08:00:13.342933 4477 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:13.344142 3939 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:00:13.344168 3790 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:00:13.344305 4338 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:00:13.344571 4436 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:13.344753 4476 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:13.344909 4477 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:13.346884 4436 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.347260 3939 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.348582 4338 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.349112 3790 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.349435 3687 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index 27 to 28, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.352797 3864 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:13.353693 3790 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:13.353775 4338 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:13.354092 4479 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:13.354430 4476 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:13.355383 4479 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.356055 3790 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.356551 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 728c3dff60a8498e99bc434ff175083f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:13.357192 4338 raft_consensus.cc:2932] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:13.357411 3684 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 28 to 29, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:13.359406 3927 tablet_service.cc:1514] Processing DeleteTablet for tablet 728c3dff60a8498e99bc434ff175083f with delete_type TABLET_DATA_TOMBSTONED (TS f818705fd267423dba3ae9ae51d3a8a1 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:47780
I20240501 08:00:13.359584 4463 tablet_replica.cc:330] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: stopping tablet replica
I20240501 08:00:13.359683 4463 raft_consensus.cc:2229] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:13.359783 4463 raft_consensus.cc:2258] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:13.360431 4463 ts_tablet_manager.cc:1875] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:13.362016 4463 ts_tablet_manager.cc:1888] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 08:00:13.362076 4463 log.cc:1196] T 728c3dff60a8498e99bc434ff175083f P f818705fd267423dba3ae9ae51d3a8a1: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/wals/728c3dff60a8498e99bc434ff175083f
I20240501 08:00:13.362416 3671 catalog_manager.cc:4830] TS f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165): tablet 728c3dff60a8498e99bc434ff175083f (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:14.363168 3862 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.364300 3939 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:14.364668 3790 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:14.364867 4488 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:14.365137 4489 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:14.366705 4479 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:14.367110 3939 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:14.368515 3790 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
W20240501 08:00:14.368943 3831 consensus_peers.cc:485] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea8a83bd3a9408e877cf926ce6bf466. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.369156 3678 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index -1 to 30, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.370666 3937 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:14.371881 3790 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:14.372480 3862 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:14.372645 4472 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:14.372929 4472 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:14.374058 4472 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.374323 3790 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.375895 3862 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.376513 3864 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:14.376477 3680 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index -1 to 29, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.377635 3790 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:14.377962 3937 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 08:00:14.378054 4479 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 08:00:14.378206 3829 consensus_peers.cc:485] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1c9c65f0c0a4a53aafc7a04061dc909. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.378371 4489 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:14.380226 4479 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
W20240501 08:00:14.381515 3905 consensus_peers.cc:485] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4711451691345009da4d87f91300ca1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.381856 3937 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:14.382130 3678 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.382391 3788 consensus_queue.cc:237] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 21, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:14.383435 3788 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
W20240501 08:00:14.384236 3754 consensus_peers.cc:485] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80c853e31bc74cb38778ed1ab133d966. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.384676 3937 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:14.385118 4422 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:14.384676 3864 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:14.385650 4422 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:14.387077 4510 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.387513 3864 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.387863 3937 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.389250 3678 catalog_manager.cc:5484] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:14.390951 3788 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:14.392242 4338 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 08:00:14.392590 4494 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:14.392892 3937 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 08:00:14.393517 4494 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:14.394492 4422 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:14.394862 4338 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:14.396663 3680 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 31 to 32, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.396755 3864 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
W20240501 08:00:14.397517 3753 consensus_peers.cc:485] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 824019b3e8674c329bdeb8467db1f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.397869 3937 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:14.397995 3939 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:00:14.398057 3788 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 08:00:14.398367 4488 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:14.398551 4479 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 08:00:14.400131 3831 consensus_peers.cc:485] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b7085039b984324870248db803ece14. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:14.400295 4488 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:14.401014 3788 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:14.402395 3678 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index -1 to 28, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.403240 3937 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:14.786612 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:14.795804 3790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
new_leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
from {username='slave'} at 127.0.0.1:52016
I20240501 08:00:14.795951 3790 raft_consensus.cc:595] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Received request to transfer leadership to TS f818705fd267423dba3ae9ae51d3a8a1
I20240501 08:00:14.799623 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:14.799763 3862 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
I20240501 08:00:14.800386 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:14.800494 3862 raft_consensus.cc:595] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
I20240501 08:00:14.801054 3788 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
new_leader_uuid: "21f956c9a93b4bf19575304199b36492"
from {username='slave'} at 127.0.0.1:52016
I20240501 08:00:14.801149 3788 raft_consensus.cc:595] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Received request to transfer leadership to TS 21f956c9a93b4bf19575304199b36492
I20240501 08:00:14.801812 3937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
mode: GRACEFUL
new_leader_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:47826
I20240501 08:00:14.801906 3937 raft_consensus.cc:595] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Received request to transfer leadership to TS d5c9f893d7344b4f841e68668e19ab97
I20240501 08:00:14.802492 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "385e9e7fb97848e69d579dd53df27aae"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:14.802608 3862 raft_consensus.cc:595] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Received request to transfer leadership to TS 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:14.802850 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240501 08:00:14.802958 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:14.831544 4526 ts_tablet_manager.cc:902] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:14.832181 4526 tablet_copy_client.cc:308] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34681
I20240501 08:00:14.833189 4532 ts_tablet_manager.cc:902] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:14.833693 4532 tablet_copy_client.cc:308] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34681
I20240501 08:00:14.834141 3876 tablet_copy_service.cc:140] P d5c9f893d7344b4f841e68668e19ab97: Received BeginTabletCopySession request for tablet e1c9c65f0c0a4a53aafc7a04061dc909 from peer 385e9e7fb97848e69d579dd53df27aae ({username='slave'} at 127.0.0.1:47650)
I20240501 08:00:14.834259 3876 tablet_copy_service.cc:161] P d5c9f893d7344b4f841e68668e19ab97: Beginning new tablet copy session on tablet e1c9c65f0c0a4a53aafc7a04061dc909 from peer 385e9e7fb97848e69d579dd53df27aae at {username='slave'} at 127.0.0.1:47650: session id = 385e9e7fb97848e69d579dd53df27aae-e1c9c65f0c0a4a53aafc7a04061dc909
I20240501 08:00:14.835265 3876 tablet_copy_source_session.cc:205] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:14.835484 4510 raft_consensus.cc:979] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f: : Instructing follower 21f956c9a93b4bf19575304199b36492 to start an election
I20240501 08:00:14.835572 4510 raft_consensus.cc:1067] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Signalling peer 21f956c9a93b4bf19575304199b36492 to start an election
I20240501 08:00:14.835970 4338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff"
dest_uuid: "21f956c9a93b4bf19575304199b36492"
from {username='slave'} at 127.0.0.1:48944
I20240501 08:00:14.836094 4338 raft_consensus.cc:482] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:14.836153 4338 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.836179 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c9c65f0c0a4a53aafc7a04061dc909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:14.836568 3876 tablet_copy_service.cc:140] P d5c9f893d7344b4f841e68668e19ab97: Received BeginTabletCopySession request for tablet 7b7085039b984324870248db803ece14 from peer e34080744cfb46bc8cb6c41de486c343 ({username='slave'} at 127.0.0.1:47674)
I20240501 08:00:14.836647 3876 tablet_copy_service.cc:161] P d5c9f893d7344b4f841e68668e19ab97: Beginning new tablet copy session on tablet 7b7085039b984324870248db803ece14 from peer e34080744cfb46bc8cb6c41de486c343 at {username='slave'} at 127.0.0.1:47674: session id = e34080744cfb46bc8cb6c41de486c343-7b7085039b984324870248db803ece14
I20240501 08:00:14.836975 4338 raft_consensus.cc:504] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:14.837363 4338 leader_election.cc:288] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [CANDIDATE]: Term 2 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:14.837450 3876 tablet_copy_source_session.cc:205] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:14.837908 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "21f956c9a93b4bf19575304199b36492" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:14.838011 3788 raft_consensus.cc:3032] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:14.838047 3788 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:14.838163 3788 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:14.838326 3788 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.838376 4532 tablet_copy_client.cc:783] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:14.838506 4532 tablet_copy_client.cc:647] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:14.838336 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7085039b984324870248db803ece14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:14.840219 4535 tablet_copy_client.cc:1002] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:14.840693 4526 tablet_copy_client.cc:783] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:14.840732 3788 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f956c9a93b4bf19575304199b36492 in term 2.
I20240501 08:00:14.840826 4526 tablet_copy_client.cc:647] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:14.841079 4303 leader_election.cc:302] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f956c9a93b4bf19575304199b36492, 504c682c767543748142f114ffa7be2f; no voters:
I20240501 08:00:14.841238 4519 raft_consensus.cc:2781] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:14.841317 4488 raft_consensus.cc:979] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:14.841389 4488 raft_consensus.cc:1067] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:14.841405 4519 raft_consensus.cc:686] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 2 LEADER]: Becoming Leader. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Running, Role: LEADER
I20240501 08:00:14.841552 4519 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:14.841894 3789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34744
I20240501 08:00:14.842000 3789 raft_consensus.cc:482] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:14.842064 3789 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.842885 4529 tablet_copy_client.cc:1002] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:14.842945 3789 raft_consensus.cc:504] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.843276 3789 leader_election.cc:288] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:14.843410 3678 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 reported cstate change: term changed from 1 to 2, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to 21f956c9a93b4bf19575304199b36492 (127.0.99.6). New cstate: current_term: 2 leader_uuid: "21f956c9a93b4bf19575304199b36492" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.843739 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:14.843822 3937 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.844480 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:14.844563 3862 raft_consensus.cc:3032] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:14.844596 3862 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:14.844662 3937 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:14.844723 3862 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.844882 3862 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.845316 3753 leader_election.cc:302] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:14.845484 4513 raft_consensus.cc:2781] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:14.845569 4513 raft_consensus.cc:686] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:14.845654 3862 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:14.845679 4513 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.847152 3678 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 1 to 2, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.847929 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824019b3e8674c329bdeb8467db1f6ff" candidate_uuid: "21f956c9a93b4bf19575304199b36492" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:14.848011 3937 raft_consensus.cc:3037] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.848753 3937 raft_consensus.cc:2454] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f956c9a93b4bf19575304199b36492 in term 2.
I20240501 08:00:14.852239 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.852344 3789 raft_consensus.cc:595] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership
I20240501 08:00:14.856707 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.856806 3789 raft_consensus.cc:595] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership
W20240501 08:00:14.857021 3673 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7b7085039b984324870248db803ece14 already in progress [suppressed 3 similar messages]
I20240501 08:00:14.857566 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.857654 3789 raft_consensus.cc:595] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership
W20240501 08:00:14.857909 3735 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7b7085039b984324870248db803ece14 already in progress
W20240501 08:00:14.857961 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7b7085039b984324870248db803ece14 already in progress
I20240501 08:00:14.860790 4542 ts_tablet_manager.cc:902] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Initiating tablet copy from peer f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:14.861346 4542 tablet_copy_client.cc:308] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:39165
I20240501 08:00:14.861701 3951 tablet_copy_service.cc:140] P f818705fd267423dba3ae9ae51d3a8a1: Received BeginTabletCopySession request for tablet a4711451691345009da4d87f91300ca1 from peer 21f956c9a93b4bf19575304199b36492 ({username='slave'} at 127.0.0.1:35784)
I20240501 08:00:14.861791 3951 tablet_copy_service.cc:161] P f818705fd267423dba3ae9ae51d3a8a1: Beginning new tablet copy session on tablet a4711451691345009da4d87f91300ca1 from peer 21f956c9a93b4bf19575304199b36492 at {username='slave'} at 127.0.0.1:35784: session id = 21f956c9a93b4bf19575304199b36492-a4711451691345009da4d87f91300ca1
I20240501 08:00:14.861974 4489 raft_consensus.cc:979] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:14.862071 4479 raft_consensus.cc:1067] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:14.862414 3789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34744
I20240501 08:00:14.862507 3789 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:14.862568 3789 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:14.862743 3951 tablet_copy_source_session.cc:205] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:14.863613 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4711451691345009da4d87f91300ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:14.863561 3789 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.863978 3789 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 3 election: Requested vote from peers f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:14.864368 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:14.864457 3937 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:14.865300 3937 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 3.
I20240501 08:00:14.865674 4542 tablet_copy_client.cc:783] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:14.865674 3753 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:14.865846 4542 tablet_copy_client.cc:647] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:14.865895 4510 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:14.865973 4510 raft_consensus.cc:686] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:14.866086 4510 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.866113 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:14.866202 3862 raft_consensus.cc:3032] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:14.866237 3862 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:14.866354 3862 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:14.866524 3862 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:14.867470 3862 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 3.
I20240501 08:00:14.867538 3678 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 2 to 3, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 3 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.868279 4545 tablet_copy_client.cc:1002] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:14.871165 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.871254 3789 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Received request to transfer leadership
I20240501 08:00:14.874902 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.875005 3789 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Received request to transfer leadership
I20240501 08:00:14.875538 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:14.875629 3789 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Received request to transfer leadership
W20240501 08:00:14.875872 3735 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 3ea8a83bd3a9408e877cf926ce6bf466 already in progress
W20240501 08:00:14.875921 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 3ea8a83bd3a9408e877cf926ce6bf466 already in progress
I20240501 08:00:14.946413 4547 ts_tablet_manager.cc:902] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Initiating tablet copy from peer 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:14.946981 4547 tablet_copy_client.cc:308] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39565
I20240501 08:00:14.947345 3800 tablet_copy_service.cc:140] P 504c682c767543748142f114ffa7be2f: Received BeginTabletCopySession request for tablet 80c853e31bc74cb38778ed1ab133d966 from peer 21f956c9a93b4bf19575304199b36492 ({username='slave'} at 127.0.0.1:52002)
I20240501 08:00:14.947434 3800 tablet_copy_service.cc:161] P 504c682c767543748142f114ffa7be2f: Beginning new tablet copy session on tablet 80c853e31bc74cb38778ed1ab133d966 from peer 21f956c9a93b4bf19575304199b36492 at {username='slave'} at 127.0.0.1:52002: session id = 21f956c9a93b4bf19575304199b36492-80c853e31bc74cb38778ed1ab133d966
I20240501 08:00:14.948274 3800 tablet_copy_source_session.cc:205] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:14.948987 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c853e31bc74cb38778ed1ab133d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:14.950910 4547 tablet_copy_client.cc:783] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:14.951049 4547 tablet_copy_client.cc:647] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:14.951926 4551 tablet_copy_client.cc:1002] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:14.953760 4513 raft_consensus.cc:979] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f: : Instructing follower f818705fd267423dba3ae9ae51d3a8a1 to start an election
I20240501 08:00:14.953864 4510 raft_consensus.cc:1067] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Signalling peer f818705fd267423dba3ae9ae51d3a8a1 to start an election
I20240501 08:00:14.954236 3937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
from {username='slave'} at 127.0.0.1:47790
I20240501 08:00:14.954355 3937 raft_consensus.cc:482] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:14.954417 3937 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.955145 3937 raft_consensus.cc:504] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:14.955457 3937 leader_election.cc:288] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 2 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:14.955703 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:14.955765 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:14.955812 3862 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.955865 3789 raft_consensus.cc:3032] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:14.955909 3789 raft_consensus.cc:729] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:14.956038 3789 consensus_queue.cc:260] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:14.956193 3789 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.956615 3862 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 2.
I20240501 08:00:14.956838 3904 leader_election.cc:302] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:14.956969 4501 raft_consensus.cc:2781] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:14.957072 3789 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 2.
I20240501 08:00:14.957157 4501 raft_consensus.cc:686] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:14.957288 4501 consensus_queue.cc:237] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:14.958617 3678 catalog_manager.cc:5484] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 1 to 2, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } }
I20240501 08:00:14.977816 4552 raft_consensus.cc:979] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1: : Instructing follower d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:14.977924 4552 raft_consensus.cc:1067] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Signalling peer d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:14.978277 3862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:59544
I20240501 08:00:14.978410 3862 raft_consensus.cc:482] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:14.978477 3862 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.979394 3862 raft_consensus.cc:504] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:14.979776 3862 leader_election.cc:288] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 2 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:14.979969 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:14.980067 3790 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.980295 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:14.980374 3937 raft_consensus.cc:3032] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:14.980403 3937 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:14.980521 3937 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:14.980685 3937 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:14.980942 3790 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 2.
I20240501 08:00:14.981175 3831 leader_election.cc:302] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:14.981328 4479 raft_consensus.cc:2781] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:14.981415 4479 raft_consensus.cc:686] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:14.981539 3937 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 2.
I20240501 08:00:14.981529 4479 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:14.983088 3678 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 1 to 2, leader changed from f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:14.987787 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:14.987887 3862 raft_consensus.cc:595] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:00:14.990983 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:14.991079 3862 raft_consensus.cc:595] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:00:14.991634 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:14.991724 3862 raft_consensus.cc:595] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Received request to transfer leadership
W20240501 08:00:14.991992 3735 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for a4711451691345009da4d87f91300ca1 already in progress
W20240501 08:00:14.992043 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for a4711451691345009da4d87f91300ca1 already in progress
I20240501 08:00:15.197743 4479 raft_consensus.cc:979] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 385e9e7fb97848e69d579dd53df27aae to start an election
I20240501 08:00:15.197896 4489 raft_consensus.cc:1067] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Signalling peer 385e9e7fb97848e69d579dd53df27aae to start an election
I20240501 08:00:15.198287 4263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a"
dest_uuid: "385e9e7fb97848e69d579dd53df27aae"
from {username='slave'} at 127.0.0.1:34886
I20240501 08:00:15.198400 4263 raft_consensus.cc:482] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:15.198454 4263 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:15.200498 4263 raft_consensus.cc:504] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:15.200950 4263 leader_election.cc:288] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:15.201215 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "385e9e7fb97848e69d579dd53df27aae" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:15.201305 3939 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:15.201272 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909fdab57ca42a0b05c41dde754743a" candidate_uuid: "385e9e7fb97848e69d579dd53df27aae" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:15.201378 3862 raft_consensus.cc:3032] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:15.201423 3862 raft_consensus.cc:729] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:15.201550 3862 consensus_queue.cc:260] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:15.201704 3862 raft_consensus.cc:3037] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:15.202064 3939 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 385e9e7fb97848e69d579dd53df27aae in term 2.
I20240501 08:00:15.202291 4226 leader_election.cc:302] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385e9e7fb97848e69d579dd53df27aae, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:15.202484 3862 raft_consensus.cc:2454] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 385e9e7fb97848e69d579dd53df27aae in term 2.
I20240501 08:00:15.202596 4554 raft_consensus.cc:2781] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:15.202812 4554 raft_consensus.cc:686] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [term 2 LEADER]: Becoming Leader. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Running, Role: LEADER
I20240501 08:00:15.202948 4554 consensus_queue.cc:237] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:15.204245 3678 catalog_manager.cc:5484] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae reported cstate change: term changed from 1 to 2, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 385e9e7fb97848e69d579dd53df27aae (127.0.99.5). New cstate: current_term: 2 leader_uuid: "385e9e7fb97848e69d579dd53df27aae" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:15.256453 3862 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:15.256884 4510 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:15.258401 4510 raft_consensus.cc:979] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: : Instructing follower d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:15.258491 4510 raft_consensus.cc:1067] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Signalling peer d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:15.259801 3939 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:15.259778 3864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:59542
I20240501 08:00:15.259975 3864 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:15.260039 3864 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Advancing to term 4
W20240501 08:00:15.260054 3755 consensus_peers.cc:485] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea8a83bd3a9408e877cf926ce6bf466. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.261031 3864 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.261220 3862 raft_consensus.cc:1226] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 FOLLOWER]: Rejecting Update request from peer 504c682c767543748142f114ffa7be2f for earlier term 3. Current term is 4. Ops: []
I20240501 08:00:15.261380 3864 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 4 election: Requested vote from peers f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165), 504c682c767543748142f114ffa7be2f (127.0.99.1:39565)
I20240501 08:00:15.261627 3939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:15.261735 3939 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:00:15.262053 4513 consensus_queue.cc:1046] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: INVALID_TERM, Last received: 3.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:15.262200 4513 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:15.262408 4513 raft_consensus.cc:3032] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:00:15.262462 4513 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:15.262595 4513 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.262691 3939 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 4.
I20240501 08:00:15.262802 4513 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:00:15.262876 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:15.262923 3829 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:15.263067 4489 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:00:15.263147 4489 raft_consensus.cc:686] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:15.263250 3937 raft_consensus.cc:1226] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Rejecting Update request from peer 504c682c767543748142f114ffa7be2f for earlier term 3. Current term is 4. Ops: [3.31-3.31]
I20240501 08:00:15.263279 4489 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.264408 3789 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 4.
I20240501 08:00:15.265527 3678 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 3 to 4, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:15.282490 3789 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 08:00:15.282858 4539 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:15.285710 3937 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 08:00:15.286379 4560 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
W20240501 08:00:15.289149 4301 consensus_peers.cc:485] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 824019b3e8674c329bdeb8467db1f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.354387 3864 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:00:15.354794 4510 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:15.356127 4513 raft_consensus.cc:979] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: : Instructing follower d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:15.356216 4510 raft_consensus.cc:1067] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Signalling peer d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:15.357285 3937 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20240501 08:00:15.357380 3755 consensus_peers.cc:485] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.357697 4510 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:15.358019 3864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b7085039b984324870248db803ece14"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:59542
I20240501 08:00:15.358139 3864 raft_consensus.cc:482] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:15.358199 3864 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.359443 3864 raft_consensus.cc:504] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.360244 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:15.360356 3789 raft_consensus.cc:3032] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:15.360357 3864 leader_election.cc:288] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 3 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:15.360410 3789 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:15.360543 3789 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.360601 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7085039b984324870248db803ece14" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:15.360699 3937 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.360752 3789 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.361449 3937 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 3.
I20240501 08:00:15.361455 3789 raft_consensus.cc:2454] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 3.
I20240501 08:00:15.361724 3829 leader_election.cc:302] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:15.361956 4489 raft_consensus.cc:2781] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:15.362038 4489 raft_consensus.cc:686] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 3 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:15.362056 3789 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:15.362167 4489 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:15.362339 4488 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:15.363530 3684 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 2 to 3, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:15.364140 4488 raft_consensus.cc:979] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:15.364212 4488 raft_consensus.cc:1067] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
W20240501 08:00:15.364948 3830 consensus_peers.cc:485] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.365134 3937 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:15.365449 4479 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:15.365566 3790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4711451691345009da4d87f91300ca1"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34744
I20240501 08:00:15.365657 3790 raft_consensus.cc:482] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:15.365705 3790 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.366828 3790 raft_consensus.cc:504] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:15.367120 3790 leader_election.cc:288] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 3 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:15.367440 3864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:15.367594 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4711451691345009da4d87f91300ca1" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:15.367686 3937 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.368573 3937 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 3.
I20240501 08:00:15.368839 3753 leader_election.cc:302] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:15.368974 4510 raft_consensus.cc:2781] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:15.369043 4510 raft_consensus.cc:686] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:15.369194 4510 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:15.369042 3864 raft_consensus.cc:3032] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:15.370038 3864 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:15.370201 3864 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:15.370419 3864 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:15.370409 3678 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 2 to 3, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 3 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:15.371263 3864 raft_consensus.cc:2454] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 3.
I20240501 08:00:15.498292 3864 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:00:15.498654 4501 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:15.500746 3790 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 08:00:15.501055 4552 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 08:00:15.625100 3937 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:15.625491 4554 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:15.627884 3864 raft_consensus.cc:1261] T b909fdab57ca42a0b05c41dde754743a P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:15.628396 4554 consensus_queue.cc:1035] T b909fdab57ca42a0b05c41dde754743a P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:15.639256 3789 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 08:00:15.639631 4479 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:15.642058 3937 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
W20240501 08:00:15.642371 3831 consensus_peers.cc:485] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea8a83bd3a9408e877cf926ce6bf466. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.642802 4488 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
W20240501 08:00:15.767068 3754 consensus_peers.cc:485] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.769367 3937 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:15.769716 4510 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:15.771919 3864 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:15.772450 4513 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:15.813108 3937 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:00:15.813488 4479 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:15.815548 3790 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240501 08:00:15.815902 3831 consensus_peers.cc:485] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:15.816066 4488 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:15.867365 4574 ts_tablet_manager.cc:902] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Initiating tablet copy from peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819)
I20240501 08:00:15.867909 4574 tablet_copy_client.cc:308] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.6:43819
I20240501 08:00:15.871984 4348 tablet_copy_service.cc:140] P 21f956c9a93b4bf19575304199b36492: Received BeginTabletCopySession request for tablet 824019b3e8674c329bdeb8467db1f6ff from peer 385e9e7fb97848e69d579dd53df27aae ({username='slave'} at 127.0.0.1:43122)
I20240501 08:00:15.872108 4348 tablet_copy_service.cc:161] P 21f956c9a93b4bf19575304199b36492: Beginning new tablet copy session on tablet 824019b3e8674c329bdeb8467db1f6ff from peer 385e9e7fb97848e69d579dd53df27aae at {username='slave'} at 127.0.0.1:43122: session id = 385e9e7fb97848e69d579dd53df27aae-824019b3e8674c329bdeb8467db1f6ff
I20240501 08:00:15.873018 4348 tablet_copy_source_session.cc:205] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:15.873751 4574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824019b3e8674c329bdeb8467db1f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:15.875643 4574 tablet_copy_client.cc:783] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:15.875808 4574 tablet_copy_client.cc:647] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:15.876679 4578 tablet_copy_client.cc:1002] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:16.098349 4583 ts_tablet_manager.cc:902] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:16.098976 4583 tablet_copy_client.cc:308] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:34681
I20240501 08:00:16.099375 3876 tablet_copy_service.cc:140] P d5c9f893d7344b4f841e68668e19ab97: Received BeginTabletCopySession request for tablet 3ea8a83bd3a9408e877cf926ce6bf466 from peer e34080744cfb46bc8cb6c41de486c343 ({username='slave'} at 127.0.0.1:47674)
I20240501 08:00:16.099480 3876 tablet_copy_service.cc:161] P d5c9f893d7344b4f841e68668e19ab97: Beginning new tablet copy session on tablet 3ea8a83bd3a9408e877cf926ce6bf466 from peer e34080744cfb46bc8cb6c41de486c343 at {username='slave'} at 127.0.0.1:47674: session id = e34080744cfb46bc8cb6c41de486c343-3ea8a83bd3a9408e877cf926ce6bf466
I20240501 08:00:16.100299 3876 tablet_copy_source_session.cc:205] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:16.100970 4583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea8a83bd3a9408e877cf926ce6bf466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:16.102792 4583 tablet_copy_client.cc:783] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:16.102932 4583 tablet_copy_client.cc:647] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:16.103773 4587 tablet_copy_client.cc:1002] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:16.803169 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:16.804924 3937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
mode: GRACEFUL
new_leader_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:47826
I20240501 08:00:16.805092 3937 raft_consensus.cc:595] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 LEADER]: Received request to transfer leadership to TS d5c9f893d7344b4f841e68668e19ab97
I20240501 08:00:16.805765 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:16.805861 3862 raft_consensus.cc:595] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
I20240501 08:00:16.806473 3789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
new_leader_uuid: "e34080744cfb46bc8cb6c41de486c343"
from {username='slave'} at 127.0.0.1:52016
I20240501 08:00:16.806591 3789 raft_consensus.cc:595] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership to TS e34080744cfb46bc8cb6c41de486c343
I20240501 08:00:16.807196 3862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:16.807293 3862 raft_consensus.cc:595] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Received request to transfer leadership to TS 504c682c767543748142f114ffa7be2f
I20240501 08:00:16.807600 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 08:00:16.807727 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:16.840502 4532 tablet_copy_client.cc:523] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:16.842207 4532 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap starting.
I20240501 08:00:16.843123 4526 tablet_copy_client.cc:523] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:16.844455 4526 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Bootstrap starting.
I20240501 08:00:16.865804 4526 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:16.866130 4526 tablet_bootstrap.cc:492] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Bootstrap complete.
I20240501 08:00:16.866325 4526 ts_tablet_manager.cc:1367] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20240501 08:00:16.866844 4526 raft_consensus.cc:348] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:16.867017 4526 raft_consensus.cc:729] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e34080744cfb46bc8cb6c41de486c343, State: Initialized, Role: LEARNER
I20240501 08:00:16.867142 4526 consensus_queue.cc:260] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:16.867506 4526 ts_tablet_manager.cc:1398] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 08:00:16.867791 4532 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:16.868036 4532 tablet_bootstrap.cc:492] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Bootstrap complete.
I20240501 08:00:16.868250 4532 ts_tablet_manager.cc:1367] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240501 08:00:16.868294 3876 tablet_copy_service.cc:342] P d5c9f893d7344b4f841e68668e19ab97: Request end of tablet copy session e34080744cfb46bc8cb6c41de486c343-7b7085039b984324870248db803ece14 received from {username='slave'} at 127.0.0.1:47674
I20240501 08:00:16.868405 3876 tablet_copy_service.cc:433] P d5c9f893d7344b4f841e68668e19ab97: ending tablet copy session e34080744cfb46bc8cb6c41de486c343-7b7085039b984324870248db803ece14 on tablet 7b7085039b984324870248db803ece14 with peer e34080744cfb46bc8cb6c41de486c343
I20240501 08:00:16.868513 4542 tablet_copy_client.cc:523] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:16.869083 4532 raft_consensus.cc:348] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:16.869271 4532 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Initialized, Role: LEARNER
I20240501 08:00:16.869606 4532 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:16.869963 4542 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Bootstrap starting.
I20240501 08:00:16.869978 4532 ts_tablet_manager.cc:1398] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:16.870505 3876 tablet_copy_service.cc:342] P d5c9f893d7344b4f841e68668e19ab97: Request end of tablet copy session 385e9e7fb97848e69d579dd53df27aae-e1c9c65f0c0a4a53aafc7a04061dc909 received from {username='slave'} at 127.0.0.1:47650
I20240501 08:00:16.870574 3876 tablet_copy_service.cc:433] P d5c9f893d7344b4f841e68668e19ab97: ending tablet copy session 385e9e7fb97848e69d579dd53df27aae-e1c9c65f0c0a4a53aafc7a04061dc909 on tablet e1c9c65f0c0a4a53aafc7a04061dc909 with peer 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:16.890591 4542 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=140 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:16.890897 4542 tablet_bootstrap.cc:492] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Bootstrap complete.
I20240501 08:00:16.891094 4542 ts_tablet_manager.cc:1367] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.004s
I20240501 08:00:16.891532 4542 raft_consensus.cc:348] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:16.891660 4542 raft_consensus.cc:729] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Initialized, Role: LEARNER
I20240501 08:00:16.891763 4542 consensus_queue.cc:260] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:16.892037 4542 ts_tablet_manager.cc:1398] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:16.892515 3951 tablet_copy_service.cc:342] P f818705fd267423dba3ae9ae51d3a8a1: Request end of tablet copy session 21f956c9a93b4bf19575304199b36492-a4711451691345009da4d87f91300ca1 received from {username='slave'} at 127.0.0.1:35784
I20240501 08:00:16.892578 3951 tablet_copy_service.cc:433] P f818705fd267423dba3ae9ae51d3a8a1: ending tablet copy session 21f956c9a93b4bf19575304199b36492-a4711451691345009da4d87f91300ca1 on tablet a4711451691345009da4d87f91300ca1 with peer 21f956c9a93b4bf19575304199b36492
I20240501 08:00:16.935011 4187 raft_consensus.cc:3037] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 1 LEARNER]: Advancing to term 3
I20240501 08:00:16.936023 4187 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 3 LEARNER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 08:00:16.936380 4489 consensus_queue.cc:1037] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:16.938056 4489 raft_consensus.cc:1050] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97: attempting to promote NON_VOTER e34080744cfb46bc8cb6c41de486c343 to VOTER
I20240501 08:00:16.938454 4489 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:16.939677 3937 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:16.939770 4187 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 3 LEARNER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:16.939806 3789 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:16.940073 4488 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:16.940194 4479 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:16.940415 4479 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:16.941941 4479 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.942432 4187 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.942497 3789 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.944361 3937 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.944612 3678 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 28 to 31, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:16.947535 3862 consensus_queue.cc:237] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:16.948454 4187 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 08:00:16.948818 4489 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:16.948906 3937 raft_consensus.cc:1261] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 08:00:16.949194 4489 consensus_queue.cc:1035] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:16.950134 4489 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P d5c9f893d7344b4f841e68668e19ab97 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.950385 4187 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.950966 3937 raft_consensus.cc:2932] T 7b7085039b984324870248db803ece14 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.951803 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b7085039b984324870248db803ece14 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:16.952229 4547 tablet_copy_client.cc:523] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:16.952682 3678 catalog_manager.cc:5484] T 7b7085039b984324870248db803ece14 P e34080744cfb46bc8cb6c41de486c343 reported cstate change: config changed from index 31 to 32, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New cstate: current_term: 3 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:16.953541 4547 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Bootstrap starting.
I20240501 08:00:16.955986 3777 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7085039b984324870248db803ece14 with delete_type TABLET_DATA_TOMBSTONED (TS 504c682c767543748142f114ffa7be2f not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:16.956527 4607 tablet_replica.cc:330] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: stopping tablet replica
I20240501 08:00:16.956660 4607 raft_consensus.cc:2229] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:16.956871 4607 raft_consensus.cc:2258] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:16.957432 4607 ts_tablet_manager.cc:1875] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:16.959308 4607 ts_tablet_manager.cc:1888] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240501 08:00:16.959368 4607 log.cc:1196] T 7b7085039b984324870248db803ece14 P 504c682c767543748142f114ffa7be2f: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/wals/7b7085039b984324870248db803ece14
I20240501 08:00:16.959805 3673 catalog_manager.cc:4830] TS 504c682c767543748142f114ffa7be2f (127.0.99.1:39565): tablet 7b7085039b984324870248db803ece14 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:16.975498 4547 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:16.975814 4547 tablet_bootstrap.cc:492] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Bootstrap complete.
I20240501 08:00:16.976004 4547 ts_tablet_manager.cc:1367] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.006s
I20240501 08:00:16.976491 4547 raft_consensus.cc:348] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:16.976624 4547 raft_consensus.cc:729] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Initialized, Role: LEARNER
I20240501 08:00:16.976758 4547 consensus_queue.cc:260] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:16.977066 4547 ts_tablet_manager.cc:1398] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 08:00:16.977504 3800 tablet_copy_service.cc:342] P 504c682c767543748142f114ffa7be2f: Request end of tablet copy session 21f956c9a93b4bf19575304199b36492-80c853e31bc74cb38778ed1ab133d966 received from {username='slave'} at 127.0.0.1:52002
I20240501 08:00:16.977582 3800 tablet_copy_service.cc:433] P 504c682c767543748142f114ffa7be2f: ending tablet copy session 21f956c9a93b4bf19575304199b36492-80c853e31bc74cb38778ed1ab133d966 on tablet 80c853e31bc74cb38778ed1ab133d966 with peer 21f956c9a93b4bf19575304199b36492
I20240501 08:00:17.003332 4589 raft_consensus.cc:979] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f: : Instructing follower e34080744cfb46bc8cb6c41de486c343 to start an election
I20240501 08:00:17.003455 4589 raft_consensus.cc:1067] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Signalling peer e34080744cfb46bc8cb6c41de486c343 to start an election
I20240501 08:00:17.003808 4187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11"
dest_uuid: "e34080744cfb46bc8cb6c41de486c343"
from {username='slave'} at 127.0.0.1:33344
I20240501 08:00:17.003934 4187 raft_consensus.cc:482] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:17.003990 4187 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.004973 4187 raft_consensus.cc:504] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:17.005439 4187 leader_election.cc:288] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [CANDIDATE]: Term 3 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:17.005648 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "e34080744cfb46bc8cb6c41de486c343" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:17.005681 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc118979132748e1a45e3e33e655ac11" candidate_uuid: "e34080744cfb46bc8cb6c41de486c343" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:17.005761 3789 raft_consensus.cc:3032] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:17.005780 3862 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.005808 3789 raft_consensus.cc:729] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:17.005934 3789 consensus_queue.cc:260] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:17.006062 3789 raft_consensus.cc:3037] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.006678 3862 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e34080744cfb46bc8cb6c41de486c343 in term 3.
I20240501 08:00:17.006942 4150 leader_election.cc:302] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, e34080744cfb46bc8cb6c41de486c343; no voters:
I20240501 08:00:17.007133 4598 raft_consensus.cc:2781] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:17.007197 4598 raft_consensus.cc:686] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 3 LEADER]: Becoming Leader. State: Replica: e34080744cfb46bc8cb6c41de486c343, State: Running, Role: LEADER
I20240501 08:00:17.007294 4598 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:17.007372 3789 raft_consensus.cc:2454] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e34080744cfb46bc8cb6c41de486c343 in term 3.
I20240501 08:00:17.008685 3680 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 reported cstate change: term changed from 2 to 3, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to e34080744cfb46bc8cb6c41de486c343 (127.0.99.4). New cstate: current_term: 3 leader_uuid: "e34080744cfb46bc8cb6c41de486c343" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:17.016636 4552 raft_consensus.cc:979] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1: : Instructing follower d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:17.016744 4501 raft_consensus.cc:1067] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 LEADER]: Signalling peer d5c9f893d7344b4f841e68668e19ab97 to start an election
I20240501 08:00:17.017120 3862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
from {username='slave'} at 127.0.0.1:59544
I20240501 08:00:17.017235 3862 raft_consensus.cc:482] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:17.017292 3862 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.018088 3862 raft_consensus.cc:504] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:17.018473 3862 leader_election.cc:288] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 3 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:17.018668 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:17.018738 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7" candidate_uuid: "d5c9f893d7344b4f841e68668e19ab97" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:17.018832 3937 raft_consensus.cc:3032] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:17.018889 3937 raft_consensus.cc:729] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:17.019042 3937 consensus_queue.cc:260] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:17.019181 3937 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.019841 3937 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 3.
I20240501 08:00:17.018834 3790 raft_consensus.cc:3037] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:17.020107 3829 leader_election.cc:302] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c9f893d7344b4f841e68668e19ab97, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:17.020287 4600 raft_consensus.cc:2781] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:17.020360 4600 raft_consensus.cc:686] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 3 LEADER]: Becoming Leader. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:17.020480 4600 consensus_queue.cc:237] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } }
I20240501 08:00:17.020767 3790 raft_consensus.cc:2454] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c9f893d7344b4f841e68668e19ab97 in term 3.
I20240501 08:00:17.022022 3680 catalog_manager.cc:5484] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: term changed from 2 to 3, leader changed from f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) to d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.032006 4263 raft_consensus.cc:1203] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 08:00:17.077010 4337 raft_consensus.cc:3037] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 1 LEARNER]: Advancing to term 3
I20240501 08:00:17.078193 4337 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 3 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.078541 4589 consensus_queue.cc:1037] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.080324 4609 raft_consensus.cc:1050] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f: attempting to promote NON_VOTER 21f956c9a93b4bf19575304199b36492 to VOTER
I20240501 08:00:17.080660 4609 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.082476 3937 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 08:00:17.082696 3862 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 08:00:17.082906 4337 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 3 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 08:00:17.082901 4616 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.083745 4609 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.083995 4609 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.085415 4609 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.086197 4338 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.087342 3862 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.088271 3684 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 reported cstate change: config changed from index 29 to 32, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.089601 3937 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.091027 3790 consensus_queue.cc:237] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.091964 4338 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:17.092382 3937 raft_consensus.cc:1261] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:17.092403 4615 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:17.092811 4615 consensus_queue.cc:1035] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:17.093807 4616 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P 504c682c767543748142f114ffa7be2f [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.094197 4338 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.094617 3937 raft_consensus.cc:2932] T a4711451691345009da4d87f91300ca1 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.096098 3673 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4711451691345009da4d87f91300ca1 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.096679 3680 catalog_manager.cc:5484] T a4711451691345009da4d87f91300ca1 P 21f956c9a93b4bf19575304199b36492 reported cstate change: config changed from index 32 to 33, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New cstate: current_term: 3 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.099011 3856 tablet_service.cc:1514] Processing DeleteTablet for tablet a4711451691345009da4d87f91300ca1 with delete_type TABLET_DATA_TOMBSTONED (TS d5c9f893d7344b4f841e68668e19ab97 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:17.099357 4619 tablet_replica.cc:330] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: stopping tablet replica
I20240501 08:00:17.099678 4619 raft_consensus.cc:2229] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:17.099874 4619 raft_consensus.cc:2258] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:17.100545 4619 ts_tablet_manager.cc:1875] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:17.102383 4619 ts_tablet_manager.cc:1888] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240501 08:00:17.102447 4619 log.cc:1196] T a4711451691345009da4d87f91300ca1 P d5c9f893d7344b4f841e68668e19ab97: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/wals/a4711451691345009da4d87f91300ca1
I20240501 08:00:17.102954 3671 catalog_manager.cc:4830] TS d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681): tablet a4711451691345009da4d87f91300ca1 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:17.119710 4488 raft_consensus.cc:979] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:17.119820 4488 raft_consensus.cc:1067] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:17.120172 3789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34744
I20240501 08:00:17.120307 3789 raft_consensus.cc:482] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:17.120369 3789 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.121264 3789 raft_consensus.cc:504] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.121598 3789 leader_election.cc:288] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819)
I20240501 08:00:17.121884 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:17.121994 3862 raft_consensus.cc:3032] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:17.122035 3862 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:17.122046 4338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "21f956c9a93b4bf19575304199b36492"
I20240501 08:00:17.122162 3862 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.122398 4338 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.122467 3862 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.123325 4338 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:17.123396 3862 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:17.123576 3754 leader_election.cc:302] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f956c9a93b4bf19575304199b36492, 504c682c767543748142f114ffa7be2f; no voters:
I20240501 08:00:17.123723 4615 raft_consensus.cc:2781] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:17.123831 4615 raft_consensus.cc:686] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:17.124043 4615 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.125458 3684 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 1 to 2, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.156325 4338 raft_consensus.cc:1203] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 08:00:17.159905 4600 raft_consensus.cc:979] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:17.160027 4488 raft_consensus.cc:1067] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Signalling peer 504c682c767543748142f114ffa7be2f to start an election
I20240501 08:00:17.160404 3789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909"
dest_uuid: "504c682c767543748142f114ffa7be2f"
from {username='slave'} at 127.0.0.1:34744
I20240501 08:00:17.160694 3789 raft_consensus.cc:482] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:17.160754 3789 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.161669 3789 raft_consensus.cc:504] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:17.162014 3789 leader_election.cc:288] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165)
I20240501 08:00:17.162310 3937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
I20240501 08:00:17.162412 3937 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.162750 3862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "504c682c767543748142f114ffa7be2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:17.162849 3862 raft_consensus.cc:3032] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:17.162894 3862 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:17.163028 3862 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:17.163208 3862 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:17.163231 3937 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:17.163468 3753 leader_election.cc:302] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c682c767543748142f114ffa7be2f, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:17.163619 4615 raft_consensus.cc:2781] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:17.163708 4615 raft_consensus.cc:686] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Leader. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:17.163833 4615 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:17.164088 3862 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504c682c767543748142f114ffa7be2f in term 2.
I20240501 08:00:17.165822 3684 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f reported cstate change: term changed from 1 to 2, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to 504c682c767543748142f114ffa7be2f (127.0.99.1). New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.452867 3790 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer e34080744cfb46bc8cb6c41de486c343: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:17.453215 4598 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:17.456146 3862 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer e34080744cfb46bc8cb6c41de486c343: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:17.456619 4624 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:17.467121 3789 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:00:17.467484 4600 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:17.470355 3937 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 08:00:17.470917 4488 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 08:00:17.601454 3937 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.601791 4615 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.603909 3862 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.604094 4263 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 1 LEARNER]: Advancing to term 2
I20240501 08:00:17.604339 4632 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.605123 4263 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.606127 4615 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.606653 4632 raft_consensus.cc:1050] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f: attempting to promote NON_VOTER 21f956c9a93b4bf19575304199b36492 to VOTER
I20240501 08:00:17.607026 4632 consensus_queue.cc:237] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 21, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.608336 3937 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.608706 4337 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.609242 4615 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.609351 4609 raft_consensus.cc:1050] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: attempting to promote NON_VOTER 385e9e7fb97848e69d579dd53df27aae to VOTER
I20240501 08:00:17.609647 4609 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:17.609836 4638 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.611155 4263 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.611380 3864 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.611402 3862 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.611721 3939 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 08:00:17.611768 4615 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.611836 4639 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.611946 4609 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.612473 4609 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:17.612803 4632 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.614359 3939 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.614744 4338 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.614756 3862 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.616360 3678 catalog_manager.cc:5484] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index 30 to 31, 21f956c9a93b4bf19575304199b36492 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.618214 3937 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.618446 3864 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.619829 3790 consensus_queue.cc:237] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 21, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:17.619846 4639 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.621623 4338 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:17.621534 3687 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 29 to 31, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.622036 4615 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.622884 3937 raft_consensus.cc:1261] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:17.623430 4632 consensus_queue.cc:1035] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.623584 4263 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.624738 4632 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.626380 3673 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80c853e31bc74cb38778ed1ab133d966 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.626374 3937 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.626596 3680 catalog_manager.cc:5484] T 80c853e31bc74cb38778ed1ab133d966 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 31 to 32, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:17.626850 4337 raft_consensus.cc:2932] T 80c853e31bc74cb38778ed1ab133d966 P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } }
I20240501 08:00:17.628242 3789 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:17.629308 3856 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c853e31bc74cb38778ed1ab133d966 with delete_type TABLET_DATA_TOMBSTONED (TS d5c9f893d7344b4f841e68668e19ab97 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:59534
I20240501 08:00:17.629601 3864 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:17.629645 4644 tablet_replica.cc:330] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: stopping tablet replica
I20240501 08:00:17.629770 4644 raft_consensus.cc:2229] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:17.630117 4644 raft_consensus.cc:2258] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:17.630749 4263 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 08:00:17.631057 4638 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.631259 4632 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 08:00:17.631920 4644 ts_tablet_manager.cc:1875] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:17.632926 4632 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.633306 4263 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.633710 3864 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:17.634130 3673 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e1c9c65f0c0a4a53aafc7a04061dc909 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:17.635056 3684 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 31 to 32, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "504c682c767543748142f114ffa7be2f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:17.637243 3927 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c9c65f0c0a4a53aafc7a04061dc909 with delete_type TABLET_DATA_TOMBSTONED (TS f818705fd267423dba3ae9ae51d3a8a1 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:47780
I20240501 08:00:17.637956 4644 ts_tablet_manager.cc:1888] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 08:00:17.638228 4644 log.cc:1196] T 80c853e31bc74cb38778ed1ab133d966 P d5c9f893d7344b4f841e68668e19ab97: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-1-root/wals/80c853e31bc74cb38778ed1ab133d966
I20240501 08:00:17.638674 3671 catalog_manager.cc:4830] TS d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681): tablet 80c853e31bc74cb38778ed1ab133d966 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:17.638715 4645 tablet_replica.cc:330] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: stopping tablet replica
I20240501 08:00:17.639036 4645 raft_consensus.cc:2229] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:17.639199 4645 raft_consensus.cc:2258] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:17.639698 4645 ts_tablet_manager.cc:1875] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:17.641505 4645 ts_tablet_manager.cc:1888] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 08:00:17.641566 4645 log.cc:1196] T e1c9c65f0c0a4a53aafc7a04061dc909 P f818705fd267423dba3ae9ae51d3a8a1: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/wals/e1c9c65f0c0a4a53aafc7a04061dc909
I20240501 08:00:17.641980 3671 catalog_manager.cc:4830] TS f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165): tablet e1c9c65f0c0a4a53aafc7a04061dc909 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:17.738343 4338 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.738771 4639 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.739521 3864 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Refusing update from remote peer 504c682c767543748142f114ffa7be2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 08:00:17.739899 4638 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:17.876987 4574 tablet_copy_client.cc:523] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:17.878612 4574 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Bootstrap starting.
I20240501 08:00:17.902976 4574 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:17.903303 4574 tablet_bootstrap.cc:492] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Bootstrap complete.
I20240501 08:00:17.903507 4574 ts_tablet_manager.cc:1367] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240501 08:00:17.904098 4574 raft_consensus.cc:348] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:17.904259 4574 raft_consensus.cc:729] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Initialized, Role: LEARNER
I20240501 08:00:17.904367 4574 consensus_queue.cc:260] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:17.904698 4574 ts_tablet_manager.cc:1398] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:17.905225 4348 tablet_copy_service.cc:342] P 21f956c9a93b4bf19575304199b36492: Request end of tablet copy session 385e9e7fb97848e69d579dd53df27aae-824019b3e8674c329bdeb8467db1f6ff received from {username='slave'} at 127.0.0.1:43122
I20240501 08:00:17.905313 4348 tablet_copy_service.cc:433] P 21f956c9a93b4bf19575304199b36492: ending tablet copy session 385e9e7fb97848e69d579dd53df27aae-824019b3e8674c329bdeb8467db1f6ff on tablet 824019b3e8674c329bdeb8467db1f6ff with peer 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:18.067517 4263 raft_consensus.cc:1203] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Deduplicated request from leader. Original: 1.32->[2.33-2.33] Dedup: 2.33->[]
I20240501 08:00:18.104040 4583 tablet_copy_client.cc:523] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:18.105582 4583 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Bootstrap starting.
I20240501 08:00:18.129868 4583 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:18.130172 4583 tablet_bootstrap.cc:492] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Bootstrap complete.
I20240501 08:00:18.130362 4583 ts_tablet_manager.cc:1367] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20240501 08:00:18.130885 4583 raft_consensus.cc:348] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:18.131014 4583 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: e34080744cfb46bc8cb6c41de486c343, State: Initialized, Role: LEARNER
I20240501 08:00:18.131122 4583 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:18.131441 4583 ts_tablet_manager.cc:1398] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:00:18.131891 3876 tablet_copy_service.cc:342] P d5c9f893d7344b4f841e68668e19ab97: Request end of tablet copy session e34080744cfb46bc8cb6c41de486c343-3ea8a83bd3a9408e877cf926ce6bf466 received from {username='slave'} at 127.0.0.1:47674
I20240501 08:00:18.131975 3876 tablet_copy_service.cc:433] P d5c9f893d7344b4f841e68668e19ab97: ending tablet copy session e34080744cfb46bc8cb6c41de486c343-3ea8a83bd3a9408e877cf926ce6bf466 on tablet 3ea8a83bd3a9408e877cf926ce6bf466 with peer e34080744cfb46bc8cb6c41de486c343
I20240501 08:00:18.149905 4187 raft_consensus.cc:1203] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 LEARNER]: Deduplicated request from leader. Original: 3.31->[4.32-4.32] Dedup: 4.32->[]
I20240501 08:00:18.677803 4653 raft_consensus.cc:1050] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492: attempting to promote NON_VOTER 385e9e7fb97848e69d579dd53df27aae to VOTER
I20240501 08:00:18.678213 4653 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 32, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:18.679514 3790 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 08:00:18.679805 4263 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 LEARNER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 08:00:18.679875 4573 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:18.680044 3937 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 08:00:18.680150 4653 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:18.680564 4653 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:18.681985 4573 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.682358 3790 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.682590 3937 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.683811 4263 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.684412 3680 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 32 to 34, 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "21f956c9a93b4bf19575304199b36492" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.688098 4337 consensus_queue.cc:237] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:18.688953 3790 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240501 08:00:18.688977 4263 raft_consensus.cc:1261] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240501 08:00:18.689275 4656 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 08:00:18.689445 4653 consensus_queue.cc:1035] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 08:00:18.690737 4656 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 21f956c9a93b4bf19575304199b36492 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.691162 3790 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.691267 4263 raft_consensus.cc:2932] T 824019b3e8674c329bdeb8467db1f6ff P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.692179 3732 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 824019b3e8674c329bdeb8467db1f6ff with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.692847 3687 catalog_manager.cc:5484] T 824019b3e8674c329bdeb8467db1f6ff P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 34 to 35, VOTER f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "21f956c9a93b4bf19575304199b36492" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } }
I20240501 08:00:18.694772 3927 tablet_service.cc:1514] Processing DeleteTablet for tablet 824019b3e8674c329bdeb8467db1f6ff with delete_type TABLET_DATA_TOMBSTONED (TS f818705fd267423dba3ae9ae51d3a8a1 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:47780
I20240501 08:00:18.695119 4664 tablet_replica.cc:330] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: stopping tablet replica
I20240501 08:00:18.695271 4664 raft_consensus.cc:2229] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.695557 4664 raft_consensus.cc:2258] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.696035 4664 ts_tablet_manager.cc:1875] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.697685 4664 ts_tablet_manager.cc:1888] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20240501 08:00:18.697746 4664 log.cc:1196] T 824019b3e8674c329bdeb8467db1f6ff P f818705fd267423dba3ae9ae51d3a8a1: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-2-root/wals/824019b3e8674c329bdeb8467db1f6ff
I20240501 08:00:18.698154 3671 catalog_manager.cc:4830] TS f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3:39165): tablet 824019b3e8674c329bdeb8467db1f6ff (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:18.705217 4666 raft_consensus.cc:1050] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: attempting to promote NON_VOTER e34080744cfb46bc8cb6c41de486c343 to VOTER
I20240501 08:00:18.705696 4666 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:18.706947 3937 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 08:00:18.706947 3790 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 08:00:18.706959 4187 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 LEARNER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 08:00:18.707346 4668 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:18.707535 4666 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:18.707685 4600 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:18.709326 4668 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.709758 3790 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.709826 4187 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.711303 3937 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.711875 3678 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f reported cstate change: config changed from index 30 to 33, e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.714579 3864 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:18.715602 4187 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 08:00:18.715657 3937 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 08:00:18.715910 4668 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:18.716346 4668 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:18.717156 4600 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.717599 4187 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.718060 3937 raft_consensus.cc:2932] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } }
I20240501 08:00:18.718554 3671 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ea8a83bd3a9408e877cf926ce6bf466 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:00:18.719378 3684 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 reported cstate change: config changed from index 33 to 34, VOTER 504c682c767543748142f114ffa7be2f (127.0.99.1) evicted. New cstate: current_term: 4 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.721494 3777 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea8a83bd3a9408e877cf926ce6bf466 with delete_type TABLET_DATA_TOMBSTONED (TS 504c682c767543748142f114ffa7be2f not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:34734
I20240501 08:00:18.721818 4672 tablet_replica.cc:330] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: stopping tablet replica
I20240501 08:00:18.722007 4672 raft_consensus.cc:2229] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:18.722151 4672 raft_consensus.cc:2258] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:18.722723 4672 ts_tablet_manager.cc:1875] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:18.724313 4672 ts_tablet_manager.cc:1888] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240501 08:00:18.724409 4672 log.cc:1196] T 3ea8a83bd3a9408e877cf926ce6bf466 P 504c682c767543748142f114ffa7be2f: Deleting WAL directory at /tmp/dist-test-taskOmwlbI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714550313795535-396-0/minicluster-data/ts-0-root/wals/3ea8a83bd3a9408e877cf926ce6bf466
I20240501 08:00:18.724867 3673 catalog_manager.cc:4830] TS 504c682c767543748142f114ffa7be2f (127.0.99.1:39565): tablet 3ea8a83bd3a9408e877cf926ce6bf466 (table test-workload [id=5f7dcaa8c7bd4b6a86a2d2b2e006f267]) successfully deleted
I20240501 08:00:18.807971 3741 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 08:00:18.809583 3864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
mode: GRACEFUL
new_leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
from {username='slave'} at 127.0.0.1:47666
I20240501 08:00:18.809726 3864 raft_consensus.cc:595] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Received request to transfer leadership to TS f818705fd267423dba3ae9ae51d3a8a1
I20240501 08:00:18.810438 3790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
new_leader_uuid: "21f956c9a93b4bf19575304199b36492"
from {username='slave'} at 127.0.0.1:52016
I20240501 08:00:18.810555 3790 raft_consensus.cc:595] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership to TS 21f956c9a93b4bf19575304199b36492
I20240501 08:00:18.811182 3790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909"
dest_uuid: "504c682c767543748142f114ffa7be2f"
mode: GRACEFUL
new_leader_uuid: "385e9e7fb97848e69d579dd53df27aae"
from {username='slave'} at 127.0.0.1:52016
I20240501 08:00:18.811280 3790 raft_consensus.cc:595] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Received request to transfer leadership to TS 385e9e7fb97848e69d579dd53df27aae
I20240501 08:00:18.811518 3741 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 08:00:18.811646 3741 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 08:00:18.843437 4638 raft_consensus.cc:979] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f: : Instructing follower 385e9e7fb97848e69d579dd53df27aae to start an election
I20240501 08:00:18.843554 4638 raft_consensus.cc:1067] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Signalling peer 385e9e7fb97848e69d579dd53df27aae to start an election
I20240501 08:00:18.843948 4263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909"
dest_uuid: "385e9e7fb97848e69d579dd53df27aae"
from {username='slave'} at 127.0.0.1:53632
I20240501 08:00:18.844090 4263 raft_consensus.cc:482] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:18.844154 4263 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:18.844930 4263 raft_consensus.cc:504] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:18.845342 4263 leader_election.cc:288] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [CANDIDATE]: Term 3 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:18.845692 3864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "385e9e7fb97848e69d579dd53df27aae" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:18.845813 3864 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:18.846534 3864 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 385e9e7fb97848e69d579dd53df27aae in term 3.
I20240501 08:00:18.846863 4226 leader_election.cc:302] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385e9e7fb97848e69d579dd53df27aae, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:18.847062 4637 raft_consensus.cc:2781] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:18.847208 4637 raft_consensus.cc:686] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 3 LEADER]: Becoming Leader. State: Replica: 385e9e7fb97848e69d579dd53df27aae, State: Running, Role: LEADER
I20240501 08:00:18.847357 4637 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:18.848882 3680 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae reported cstate change: term changed from 2 to 3, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to 385e9e7fb97848e69d579dd53df27aae (127.0.99.5). New cstate: current_term: 3 leader_uuid: "385e9e7fb97848e69d579dd53df27aae" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:18.849001 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c9c65f0c0a4a53aafc7a04061dc909" candidate_uuid: "385e9e7fb97848e69d579dd53df27aae" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:18.849112 3790 raft_consensus.cc:3032] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:18.849159 3790 raft_consensus.cc:729] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:18.849287 3790 consensus_queue.cc:260] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } }
I20240501 08:00:18.849427 3790 raft_consensus.cc:3037] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:18.850292 3790 raft_consensus.cc:2454] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 385e9e7fb97848e69d579dd53df27aae in term 3.
I20240501 08:00:19.232432 4665 raft_consensus.cc:979] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97: : Instructing follower f818705fd267423dba3ae9ae51d3a8a1 to start an election
I20240501 08:00:19.232554 4665 raft_consensus.cc:1067] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Signalling peer f818705fd267423dba3ae9ae51d3a8a1 to start an election
I20240501 08:00:19.232939 3937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466"
dest_uuid: "f818705fd267423dba3ae9ae51d3a8a1"
from {username='slave'} at 127.0.0.1:47796
I20240501 08:00:19.233088 3937 raft_consensus.cc:482] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:19.233150 3937 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:00:19.233983 3937 raft_consensus.cc:504] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:19.234314 3937 leader_election.cc:288] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 5 election: Requested vote from peers d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681), e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005)
I20240501 08:00:19.234587 4187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "e34080744cfb46bc8cb6c41de486c343"
I20240501 08:00:19.234587 3864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea8a83bd3a9408e877cf926ce6bf466" candidate_uuid: "f818705fd267423dba3ae9ae51d3a8a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:19.234771 4187 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:00:19.234797 3864 raft_consensus.cc:3032] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:00:19.234834 3864 raft_consensus.cc:729] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d5c9f893d7344b4f841e68668e19ab97, State: Running, Role: LEADER
I20240501 08:00:19.234969 3864 consensus_queue.cc:260] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:19.235123 3864 raft_consensus.cc:3037] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:00:19.235684 4187 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 5.
I20240501 08:00:19.235924 3864 raft_consensus.cc:2454] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f818705fd267423dba3ae9ae51d3a8a1 in term 5.
I20240501 08:00:19.235966 3906 leader_election.cc:302] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e34080744cfb46bc8cb6c41de486c343, f818705fd267423dba3ae9ae51d3a8a1; no voters:
I20240501 08:00:19.236168 4501 raft_consensus.cc:2781] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:00:19.236315 4501 raft_consensus.cc:686] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [term 5 LEADER]: Becoming Leader. State: Replica: f818705fd267423dba3ae9ae51d3a8a1, State: Running, Role: LEADER
I20240501 08:00:19.236439 4501 consensus_queue.cc:237] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:19.237720 3680 catalog_manager.cc:5484] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: term changed from 4 to 5, leader changed from d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2) to f818705fd267423dba3ae9ae51d3a8a1 (127.0.99.3). New cstate: current_term: 5 leader_uuid: "f818705fd267423dba3ae9ae51d3a8a1" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:00:19.288893 3790 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:19.289266 4637 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:19.291577 3864 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 08:00:19.292008 4637 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 08:00:19.333957 4673 raft_consensus.cc:979] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f: : Instructing follower 21f956c9a93b4bf19575304199b36492 to start an election
I20240501 08:00:19.334096 4673 raft_consensus.cc:1067] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Signalling peer 21f956c9a93b4bf19575304199b36492 to start an election
I20240501 08:00:19.334511 4337 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f"
dest_uuid: "21f956c9a93b4bf19575304199b36492"
from {username='slave'} at 127.0.0.1:48944
I20240501 08:00:19.334679 4337 raft_consensus.cc:482] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:19.334759 4337 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:19.335628 4337 raft_consensus.cc:504] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:19.336042 4337 leader_election.cc:288] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [CANDIDATE]: Term 3 election: Requested vote from peers 504c682c767543748142f114ffa7be2f (127.0.99.1:39565), d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
I20240501 08:00:19.336314 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "21f956c9a93b4bf19575304199b36492" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "504c682c767543748142f114ffa7be2f"
I20240501 08:00:19.336390 3864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728c3dff60a8498e99bc434ff175083f" candidate_uuid: "21f956c9a93b4bf19575304199b36492" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "d5c9f893d7344b4f841e68668e19ab97"
I20240501 08:00:19.336428 3790 raft_consensus.cc:3032] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:19.336464 3790 raft_consensus.cc:729] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 504c682c767543748142f114ffa7be2f, State: Running, Role: LEADER
I20240501 08:00:19.336475 3864 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:19.336617 3790 consensus_queue.cc:260] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:19.336787 3790 raft_consensus.cc:3037] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:19.337471 3864 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f956c9a93b4bf19575304199b36492 in term 3.
I20240501 08:00:19.337582 3790 raft_consensus.cc:2454] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f956c9a93b4bf19575304199b36492 in term 3.
I20240501 08:00:19.337754 4301 leader_election.cc:302] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f956c9a93b4bf19575304199b36492, d5c9f893d7344b4f841e68668e19ab97; no voters:
I20240501 08:00:19.337913 4653 raft_consensus.cc:2781] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:19.338078 4653 raft_consensus.cc:686] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [term 3 LEADER]: Becoming Leader. State: Replica: 21f956c9a93b4bf19575304199b36492, State: Running, Role: LEADER
I20240501 08:00:19.338194 4653 consensus_queue.cc:237] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
I20240501 08:00:19.339517 3680 catalog_manager.cc:5484] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 reported cstate change: term changed from 2 to 3, leader changed from 504c682c767543748142f114ffa7be2f (127.0.99.1) to 21f956c9a93b4bf19575304199b36492 (127.0.99.6). New cstate: current_term: 3 leader_uuid: "21f956c9a93b4bf19575304199b36492" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:00:19.681629 4187 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P e34080744cfb46bc8cb6c41de486c343 [term 5 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 08:00:19.682011 4501 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 08:00:19.684585 3864 raft_consensus.cc:1261] T 3ea8a83bd3a9408e877cf926ce6bf466 P d5c9f893d7344b4f841e68668e19ab97 [term 5 FOLLOWER]: Refusing update from remote peer f818705fd267423dba3ae9ae51d3a8a1: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 08:00:19.684937 4679 consensus_queue.cc:1035] T 3ea8a83bd3a9408e877cf926ce6bf466 P f818705fd267423dba3ae9ae51d3a8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 08:00:19.698833 4187 consensus_queue.cc:237] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } }
I20240501 08:00:19.700091 3790 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer e34080744cfb46bc8cb6c41de486c343: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 08:00:19.700373 3864 raft_consensus.cc:1261] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer e34080744cfb46bc8cb6c41de486c343: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 08:00:19.700500 4694 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:19.700667 4651 consensus_queue.cc:1035] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:19.702149 4694 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:19.702836 3790 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:19.702905 3864 raft_consensus.cc:2932] T dc118979132748e1a45e3e33e655ac11 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } } }
I20240501 08:00:19.704710 3687 catalog_manager.cc:5484] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 reported cstate change: config changed from index 32 to 34, NON_VOTER 385e9e7fb97848e69d579dd53df27aae (127.0.99.5) added. New cstate: current_term: 3 leader_uuid: "e34080744cfb46bc8cb6c41de486c343" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:19.704898 4150 consensus_peers.cc:485] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343 -> Peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185): Couldn't send request to peer 385e9e7fb97848e69d579dd53df27aae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc118979132748e1a45e3e33e655ac11. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.708484 4263 consensus_queue.cc:237] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } }
I20240501 08:00:19.709743 3864 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 08:00:19.709935 3790 raft_consensus.cc:1261] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer 385e9e7fb97848e69d579dd53df27aae: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 08:00:19.710086 4699 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:19.710350 4676 consensus_queue.cc:1035] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 08:00:19.711711 4676 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:19.711994 3864 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:19.713812 3790 raft_consensus.cc:2932] T e1c9c65f0c0a4a53aafc7a04061dc909 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } } }
I20240501 08:00:19.713830 3684 catalog_manager.cc:5484] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae reported cstate change: config changed from index 32 to 34, NON_VOTER e34080744cfb46bc8cb6c41de486c343 (127.0.99.4) added. New cstate: current_term: 3 leader_uuid: "385e9e7fb97848e69d579dd53df27aae" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385e9e7fb97848e69d579dd53df27aae" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42185 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:00:19.715124 4228 consensus_peers.cc:485] T e1c9c65f0c0a4a53aafc7a04061dc909 P 385e9e7fb97848e69d579dd53df27aae -> Peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005): Couldn't send request to peer e34080744cfb46bc8cb6c41de486c343. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1c9c65f0c0a4a53aafc7a04061dc909. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.715471 3864 consensus_queue.cc:237] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } }
I20240501 08:00:19.717096 3937 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240501 08:00:19.717339 3830 consensus_peers.cc:485] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 -> Peer 21f956c9a93b4bf19575304199b36492 (127.0.99.6:43819): Couldn't send request to peer 21f956c9a93b4bf19575304199b36492. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:19.717427 4665 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:19.717489 3790 raft_consensus.cc:1261] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer d5c9f893d7344b4f841e68668e19ab97: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 08:00:19.717790 4665 consensus_queue.cc:1035] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 08:00:19.718969 4678 raft_consensus.cc:2932] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P d5c9f893d7344b4f841e68668e19ab97 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:19.719286 3937 raft_consensus.cc:2932] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:19.720639 3790 raft_consensus.cc:2932] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
I20240501 08:00:19.721060 3687 catalog_manager.cc:5484] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P f818705fd267423dba3ae9ae51d3a8a1 reported cstate change: config changed from index -1 to 30, NON_VOTER 21f956c9a93b4bf19575304199b36492 (127.0.99.6) added. New cstate: current_term: 3 leader_uuid: "d5c9f893d7344b4f841e68668e19ab97" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } attrs { replace: true } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: true } } }
W20240501 08:00:19.722157 3735 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 08:00:19.725366 3735 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 728c3dff60a8498e99bc434ff175083f, TS 504c682c767543748142f114ffa7be2f -> TS 385e9e7fb97848e69d579dd53df27aae move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
W20240501 08:00:19.725415 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 728c3dff60a8498e99bc434ff175083f, TS 504c682c767543748142f114ffa7be2f -> TS 385e9e7fb97848e69d579dd53df27aae move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
W20240501 08:00:19.728150 3735 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a4711451691345009da4d87f91300ca1, TS f818705fd267423dba3ae9ae51d3a8a1 -> TS e34080744cfb46bc8cb6c41de486c343 move failed, destination replica disappeared from tablet's Raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
W20240501 08:00:19.728199 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet a4711451691345009da4d87f91300ca1, TS f818705fd267423dba3ae9ae51d3a8a1 -> TS e34080744cfb46bc8cb6c41de486c343 move failed, destination replica disappeared from tablet's Raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "21f956c9a93b4bf19575304199b36492" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 43819 } attrs { promote: false } }
W20240501 08:00:19.730994 3735 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f1ef543279a8493295c4439e6d8d87cd, TS d5c9f893d7344b4f841e68668e19ab97 -> TS 21f956c9a93b4bf19575304199b36492 move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
W20240501 08:00:19.731041 3735 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet f1ef543279a8493295c4439e6d8d87cd, TS d5c9f893d7344b4f841e68668e19ab97 -> TS 21f956c9a93b4bf19575304199b36492 move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 } } peers { permanent_uuid: "f818705fd267423dba3ae9ae51d3a8a1" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 39165 } } peers { permanent_uuid: "e34080744cfb46bc8cb6c41de486c343" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 43005 } attrs { promote: false } }
I20240501 08:00:19.825304 3790 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P 504c682c767543748142f114ffa7be2f [term 3 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:19.825699 4653 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c682c767543748142f114ffa7be2f" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:19.827992 3864 raft_consensus.cc:1261] T 728c3dff60a8498e99bc434ff175083f P d5c9f893d7344b4f841e68668e19ab97 [term 3 FOLLOWER]: Refusing update from remote peer 21f956c9a93b4bf19575304199b36492: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 08:00:19.828368 4685 consensus_queue.cc:1035] T 728c3dff60a8498e99bc434ff175083f P 21f956c9a93b4bf19575304199b36492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c9f893d7344b4f841e68668e19ab97" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 08:00:20.241322 4710 ts_tablet_manager.cc:902] T dc118979132748e1a45e3e33e655ac11 P 385e9e7fb97848e69d579dd53df27aae: Initiating tablet copy from peer e34080744cfb46bc8cb6c41de486c343 (127.0.99.4:43005)
I20240501 08:00:20.241855 4710 tablet_copy_client.cc:308] T dc118979132748e1a45e3e33e655ac11 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.4:43005
I20240501 08:00:20.242249 4197 tablet_copy_service.cc:140] P e34080744cfb46bc8cb6c41de486c343: Received BeginTabletCopySession request for tablet dc118979132748e1a45e3e33e655ac11 from peer 385e9e7fb97848e69d579dd53df27aae ({username='slave'} at 127.0.0.1:37682)
I20240501 08:00:20.242358 4197 tablet_copy_service.cc:161] P e34080744cfb46bc8cb6c41de486c343: Beginning new tablet copy session on tablet dc118979132748e1a45e3e33e655ac11 from peer 385e9e7fb97848e69d579dd53df27aae at {username='slave'} at 127.0.0.1:37682: session id = 385e9e7fb97848e69d579dd53df27aae-dc118979132748e1a45e3e33e655ac11
I20240501 08:00:20.243307 4197 tablet_copy_source_session.cc:205] T dc118979132748e1a45e3e33e655ac11 P e34080744cfb46bc8cb6c41de486c343: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:20.244064 4710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc118979132748e1a45e3e33e655ac11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:20.246079 4710 tablet_copy_client.cc:783] T dc118979132748e1a45e3e33e655ac11 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:20.246225 4710 tablet_copy_client.cc:647] T dc118979132748e1a45e3e33e655ac11 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:20.247125 4712 tablet_copy_client.cc:1002] T dc118979132748e1a45e3e33e655ac11 P 385e9e7fb97848e69d579dd53df27aae: tablet copy: Injecting latency into file download: 2000
I20240501 08:00:20.293613 4715 ts_tablet_manager.cc:902] T 1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7 P 21f956c9a93b4bf19575304199b36492: Initiating tablet copy from peer d5c9f893d7344b4f841e68668e19ab97 (127.0.99.2:34681)
F20240501 08:00:20.293829 4715 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.queue_time_us, tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.queue_time_us) (tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.run_cpu_time_us, tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.run_cpu_time_us) (tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.run_wall_time_us, tablet-download-pool-04e93b62b07e4f78a4afddc468b359b1.run_wall_time_us) (tablet-download-pool-1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7.queue_time_us, tablet-download-pool-1a5ce0b5fa8e44d7bf1e1c9a9cdb38d7.queue_time_us) (tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.queue_time_us, tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.queue_time_us) (tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.run_cpu_time_us, tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.run_cpu_time_us) (tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.run_wall_time_us, tablet-download-pool-3ea8a83bd3a9408e877cf926ce6bf466.run_wall_time_us) (tablet-download-pool-728c3dff60a8498e99bc434ff175083f.queue_time_us, tablet-download-pool-728c3dff60a8498e99bc434ff175083f.queue_time_us) (tablet-download-pool-728c3dff60a8498e99bc434ff175083f.run_cpu_time_us, tablet-download-pool-728c3dff60a8498e99bc434ff175083f.run_cpu_time_us) (tablet-download-pool-728c3dff60a8498e99bc434ff175083f.run_wall_time_us, tablet-download-pool-728c3dff60a8498e99bc434ff175083f.run_wall_time_us) (tablet-download-pool-7b7085039b984324870248db803ece14.queue_time_us, tablet-download-pool-7b7085039b984324870248db803ece14.queue_time_us) (tablet-download-pool-7b7085039b984324870248db803ece14.run_cpu_time_us, tablet-download-pool-7b7085039b984324870248db803ece14.run_cpu_time_us) (tablet-download-pool-7b7085039b984324870248db803ece14.run_wall_time_us, tablet-download-pool-7b7085039b984324870248db803ece14.run_wall_time_us) (tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.queue_time_us, tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.queue_time_us) (tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.run_cpu_time_us, tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.run_cpu_time_us) (tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.run_wall_time_us, tablet-download-pool-80c853e31bc74cb38778ed1ab133d966.run_wall_time_us) (tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.queue_time_us, tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.queue_time_us) (tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.run_cpu_time_us, tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.run_cpu_time_us) (tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.run_wall_time_us, tablet-download-pool-824019b3e8674c329bdeb8467db1f6ff.run_wall_time_us) (tablet-download-pool-9d827adaf117470fa9fada0d8f245671.queue_time_us, tablet-download-pool-9d827adaf117470fa9fada0d8f245671.queue_time_us) (tablet-download-pool-9d827adaf117470fa9fada0d8f245671.run_cpu_time_us, tablet-download-pool-9d827adaf117470fa9fada0d8f245671.run_cpu_time_us) (tablet-download-pool-9d827adaf117470fa9fada0d8f245671.run_wall_time_us, tablet-download-pool-9d827adaf117470fa9fada0d8f245671.run_wall_time_us) (tablet-download-pool-a4711451691345009da4d87f91300ca1.queue_time_us, tablet-download-pool-a4711451691345009da4d87f91300ca1.queue_time_us) (tablet-download-pool-a4711451691345009da4d87f91300ca1.run_cpu_time_us, tablet-download-pool-a4711451691345009da4d87f91300ca1.run_cpu_time_us) (tablet-download-pool-a4711451691345009da4d87f91300ca1.run_wall_time_us, tablet-download-pool-a4711451691345009da4d87f91300ca1.run_wall_time_us) (tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.queue_time_us, tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.queue_time_us) (tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.run_cpu_time_us, tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.run_cpu_time_us) (tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.run_wall_time_us, tablet-download-pool-b909fdab57ca42a0b05c41dde754743a.run_wall_time_us) (tablet-download-pool-dc118979132748e1a45e3e33e655ac11.queue_time_us, tablet-download-pool-dc118979132748e1a45e3e33e655ac11.queue_time_us) (tablet-download-pool-dc118979132748e1a45e3e33e655ac11.run_cpu_time_us, tablet-download-pool-dc118979132748e1a45e3e33e655ac11.run_cpu_time_us) (tablet-download-pool-dc118979132748e1a45e3e33e655ac11.run_wall_time_us, tablet-download-pool-dc118979132748e1a45e3e33e655ac11.run_wall_time_us) (tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.queue_time_us, tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.queue_time_us) (tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.run_cpu_time_us, tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.run_cpu_time_us) (tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.run_wall_time_us, tablet-download-pool-e1c9c65f0c0a4a53aafc7a04061dc909.run_wall_time_us) (tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.queue_time_us, tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.queue_time_us) (tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.run_cpu_time_us, tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.run_cpu_time_us) (tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.run_wall_time_us, tablet-download-pool-f1ef543279a8493295c4439e6d8d87cd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714550420 (unix time) try "date -d @1714550420" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7f16a5d21700) from PID 396; stack trace: ***
@ 0x7f17537c2040 (unknown) at ??:0
@ 0x7f17537c1fb7 gsignal at ??:0
@ 0x7f17537c3921 abort at ??:0
I20240501 08:00:20.313637 4719 ts_tablet_manager.cc:902] T e1c9c65f0c0a4a53aafc7a04061dc909 P e34080744cfb46bc8cb6c41de486c343: Initiating tablet copy from peer 385e9e7fb97848e69d579dd53df27aae (127.0.99.5:42185)
@ 0x7f17544c6dcd google::LogMessage::Fail() at ??:0
@ 0x7f17544cab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f17544c67cc google::LogMessage::Flush() at ??:0
@ 0x7f17544c7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f175501e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f1755002f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f1755000efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f1755f7d77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f1755f86a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f175600776f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f175601b4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f1756006226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f1756015c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f17565703cc std::function<>::operator()() at ??:0
@ 0x7f1755005e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f17550066ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f175500803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f17565703cc std::function<>::operator()() at ??:0
@ 0x7f1754ff5807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f17517f76db start_thread at ??:0
@ 0x7f17538a471f clone at ??:0