[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:05:59.585866 12500 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.53.62:42995
I20240503 02:05:59.587237 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:05:59.588353 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:59.604153 12509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:59.604575 12507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:59.605669 12500 server_base.cc:1034] running on GCE node
W20240503 02:05:59.605182 12506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:00.650741 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:00.650987 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:00.651146 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701960651130 us; error 0 us; skew 500 ppm
I20240503 02:06:00.651916 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:00.658174 12500 webserver.cc:460] Webserver started at http://127.12.53.62:33041/ using document root <none> and password file <none>
I20240503 02:06:00.659060 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:00.659274 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:00.659680 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:00.665922 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/master-0-root/instance:
uuid: "66f348d5cb6048c5b3f7652e70c4076b"
format_stamp: "Formatted at 2024-05-03 02:06:00 on dist-test-slave-fmt4"
I20240503 02:06:00.673605 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240503 02:06:00.678653 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:00.679790 12500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240503 02:06:00.680152 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/master-0-root
uuid: "66f348d5cb6048c5b3f7652e70c4076b"
format_stamp: "Formatted at 2024-05-03 02:06:00 on dist-test-slave-fmt4"
I20240503 02:06:00.680507 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:00.734639 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:00.736151 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:06:00.736596 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:00.804044 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.62:42995
I20240503 02:06:00.804144 12566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.62:42995 every 8 connection(s)
I20240503 02:06:00.810951 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:00.834694 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: Bootstrap starting.
I20240503 02:06:00.840217 12567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:00.842072 12567 log.cc:824] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:06:00.846575 12567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: No bootstrap required, opened a new log
I20240503 02:06:00.868799 12567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER }
I20240503 02:06:00.869374 12567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:00.869588 12567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f348d5cb6048c5b3f7652e70c4076b, State: Initialized, Role: FOLLOWER
I20240503 02:06:00.870318 12567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER }
I20240503 02:06:00.870810 12567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:06:00.871052 12567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:06:00.871347 12567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:00.875435 12567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER }
I20240503 02:06:00.876341 12567 leader_election.cc:302] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f348d5cb6048c5b3f7652e70c4076b; no voters:
I20240503 02:06:00.878211 12567 leader_election.cc:288] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:06:00.878573 12570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:00.880684 12570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 1 LEADER]: Becoming Leader. State: Replica: 66f348d5cb6048c5b3f7652e70c4076b, State: Running, Role: LEADER
I20240503 02:06:00.881605 12570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER }
I20240503 02:06:00.882645 12567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:06:00.891258 12572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f348d5cb6048c5b3f7652e70c4076b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER } }
I20240503 02:06:00.892117 12572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:00.892581 12571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f348d5cb6048c5b3f7652e70c4076b. Latest consensus state: current_term: 1 leader_uuid: "66f348d5cb6048c5b3f7652e70c4076b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f348d5cb6048c5b3f7652e70c4076b" member_type: VOTER } }
I20240503 02:06:00.893253 12571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:00.895438 12579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:06:00.906697 12579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:06:00.908593 12500 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:06:00.923988 12579 catalog_manager.cc:1347] Generated new cluster ID: 615b04f0792842ad9ca15b3a70b53463
I20240503 02:06:00.924366 12579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:06:00.946401 12579 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:06:00.947994 12579 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:06:00.965075 12579 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: Generated new TSK 0
I20240503 02:06:00.966183 12579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:06:00.975239 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:00.983871 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:00.988324 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:00.988988 12592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:00.989780 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:00.990481 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:00.990670 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:00.990825 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701960990806 us; error 0 us; skew 500 ppm
I20240503 02:06:00.991449 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:00.995569 12500 webserver.cc:460] Webserver started at http://127.12.53.1:39469/ using document root <none> and password file <none>
I20240503 02:06:00.996157 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:00.996357 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:00.996629 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:00.997879 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-0-root/instance:
uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
format_stamp: "Formatted at 2024-05-03 02:06:00 on dist-test-slave-fmt4"
I20240503 02:06:01.003697 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:06:01.007913 12597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.008718 12500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 02:06:01.009063 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
format_stamp: "Formatted at 2024-05-03 02:06:00 on dist-test-slave-fmt4"
I20240503 02:06:01.009361 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:01.028141 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:01.029450 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:01.032078 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:01.036533 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:01.036799 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 02:06:01.037089 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:01.037257 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.132388 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.1:40119
I20240503 02:06:01.132488 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.1:40119 every 8 connection(s)
I20240503 02:06:01.139288 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:01.148080 12664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:01.158751 12665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:01.159879 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:01.160773 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:01.162099 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:01.162397 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:01.162611 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701961162593 us; error 0 us; skew 500 ppm
I20240503 02:06:01.163246 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:01.163781 12660 heartbeater.cc:347] Connected to a master server at 127.12.53.62:42995
I20240503 02:06:01.164233 12660 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:01.165591 12660 heartbeater.cc:511] Master 127.12.53.62:42995 requested a full tablet report, sending...
I20240503 02:06:01.167730 12500 webserver.cc:460] Webserver started at http://127.12.53.2:37621/ using document root <none> and password file <none>
I20240503 02:06:01.168226 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:01.168383 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:01.168591 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:01.169081 12532 ts_manager.cc:194] Registered new tserver with Master: 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:01.169836 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-1-root/instance:
uuid: "3cd93fb4924f4c19ac7114f63d136988"
format_stamp: "Formatted at 2024-05-03 02:06:01 on dist-test-slave-fmt4"
I20240503 02:06:01.172199 12532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42270
I20240503 02:06:01.175308 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 02:06:01.178956 12672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.179769 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:06:01.180043 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
uuid: "3cd93fb4924f4c19ac7114f63d136988"
format_stamp: "Formatted at 2024-05-03 02:06:01 on dist-test-slave-fmt4"
I20240503 02:06:01.180322 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:01.196439 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:01.197719 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:01.199997 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:01.203387 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:01.203583 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.203816 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:01.203977 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.295990 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.2:40495
I20240503 02:06:01.296108 12734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.2:40495 every 8 connection(s)
I20240503 02:06:01.301980 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:01.312911 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:01.314018 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:01.315639 12735 heartbeater.cc:347] Connected to a master server at 127.12.53.62:42995
I20240503 02:06:01.316217 12735 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:01.317414 12735 heartbeater.cc:511] Master 127.12.53.62:42995 requested a full tablet report, sending...
I20240503 02:06:01.319769 12532 ts_manager.cc:194] Registered new tserver with Master: 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
W20240503 02:06:01.320602 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:01.321285 12532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42284
I20240503 02:06:01.321769 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:01.322841 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:01.323153 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:01.323385 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701961323369 us; error 0 us; skew 500 ppm
I20240503 02:06:01.324137 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:01.326989 12500 webserver.cc:460] Webserver started at http://127.12.53.3:40923/ using document root <none> and password file <none>
I20240503 02:06:01.327600 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:01.327836 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:01.328161 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:01.329562 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-2-root/instance:
uuid: "8a4ef27ae015452da6e963fea8a35bed"
format_stamp: "Formatted at 2024-05-03 02:06:01 on dist-test-slave-fmt4"
I20240503 02:06:01.335279 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240503 02:06:01.339440 12746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.340289 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:06:01.340653 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
uuid: "8a4ef27ae015452da6e963fea8a35bed"
format_stamp: "Formatted at 2024-05-03 02:06:01 on dist-test-slave-fmt4"
I20240503 02:06:01.341080 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714701959428988-12500-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:01.396688 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:01.397960 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:01.399333 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:01.401549 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:01.401752 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.401971 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:01.402120 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:01.486150 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.3:46457
I20240503 02:06:01.486248 12808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.3:46457 every 8 connection(s)
I20240503 02:06:01.502777 12809 heartbeater.cc:347] Connected to a master server at 127.12.53.62:42995
I20240503 02:06:01.503146 12809 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:01.503841 12809 heartbeater.cc:511] Master 127.12.53.62:42995 requested a full tablet report, sending...
I20240503 02:06:01.506423 12532 ts_manager.cc:194] Registered new tserver with Master: 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:01.506999 12500 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013367355s
I20240503 02:06:01.507642 12532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42288
I20240503 02:06:01.510041 12500 test_util.cc:268] Using random seed: 693091454
I20240503 02:06:01.560528 12532 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42292:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:01.564131 12532 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:01.678529 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 05f402c817de4a92874891a7efaeb2cf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 02:06:01.679379 12768 tablet_service.cc:1467] Processing CreateTablet for tablet 68e55ac373a747e6bb8a5ebbd7ca120f (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 02:06:01.675877 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 83394d7a6987493fbe49151753f5df90 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 02:06:01.680166 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f402c817de4a92874891a7efaeb2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.673597 12772 tablet_service.cc:1467] Processing CreateTablet for tablet 4eabecfb83454bb2936a70288a4cb5bf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 02:06:01.681545 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 33d69aa35cb34fa78e7f44ac709e64c4 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 02:06:01.685221 12771 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea486dc5bc4846b8e5317ea57ffbc8 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 02:06:01.687448 12767 tablet_service.cc:1467] Processing CreateTablet for tablet e3eb4bfbb3a54b4aade2249ab5390b60 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 02:06:01.689935 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e55ac373a747e6bb8a5ebbd7ca120f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.689571 12774 tablet_service.cc:1467] Processing CreateTablet for tablet 0afaeb28c1164547bf2677d565338ca5 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 02:06:01.692276 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eb4bfbb3a54b4aade2249ab5390b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.695765 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d69aa35cb34fa78e7f44ac709e64c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.701308 12774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afaeb28c1164547bf2677d565338ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.700006 12625 tablet_service.cc:1467] Processing CreateTablet for tablet 0afaeb28c1164547bf2677d565338ca5 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 02:06:01.696956 12766 tablet_service.cc:1467] Processing CreateTablet for tablet 31d74183a34b4ff2a6300a506140dbe6 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 02:06:01.701323 12624 tablet_service.cc:1467] Processing CreateTablet for tablet 83394d7a6987493fbe49151753f5df90 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 02:06:01.702586 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea486dc5bc4846b8e5317ea57ffbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.703198 12623 tablet_service.cc:1467] Processing CreateTablet for tablet 4eabecfb83454bb2936a70288a4cb5bf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 02:06:01.706326 12622 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea486dc5bc4846b8e5317ea57ffbc8 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 02:06:01.698897 12700 tablet_service.cc:1467] Processing CreateTablet for tablet 0afaeb28c1164547bf2677d565338ca5 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 02:06:01.709687 12699 tablet_service.cc:1467] Processing CreateTablet for tablet 83394d7a6987493fbe49151753f5df90 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 02:06:01.712622 12699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83394d7a6987493fbe49151753f5df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.713896 12625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afaeb28c1164547bf2677d565338ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.716043 12697 tablet_service.cc:1467] Processing CreateTablet for tablet f7ea486dc5bc4846b8e5317ea57ffbc8 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 02:06:01.721176 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83394d7a6987493fbe49151753f5df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.715981 12621 tablet_service.cc:1467] Processing CreateTablet for tablet 33d69aa35cb34fa78e7f44ac709e64c4 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 02:06:01.715521 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 4eabecfb83454bb2936a70288a4cb5bf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 02:06:01.723762 12620 tablet_service.cc:1467] Processing CreateTablet for tablet 05f402c817de4a92874891a7efaeb2cf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 02:06:01.728396 12700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afaeb28c1164547bf2677d565338ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.726611 12622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea486dc5bc4846b8e5317ea57ffbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.725384 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 33d69aa35cb34fa78e7f44ac709e64c4 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 02:06:01.732918 12619 tablet_service.cc:1467] Processing CreateTablet for tablet 68e55ac373a747e6bb8a5ebbd7ca120f (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 02:06:01.736673 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eabecfb83454bb2936a70288a4cb5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.738145 12694 tablet_service.cc:1467] Processing CreateTablet for tablet 68e55ac373a747e6bb8a5ebbd7ca120f (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 02:06:01.739498 12693 tablet_service.cc:1467] Processing CreateTablet for tablet e3eb4bfbb3a54b4aade2249ab5390b60 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 02:06:01.731657 12695 tablet_service.cc:1467] Processing CreateTablet for tablet 05f402c817de4a92874891a7efaeb2cf (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 02:06:01.740218 12618 tablet_service.cc:1467] Processing CreateTablet for tablet e3eb4bfbb3a54b4aade2249ab5390b60 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 02:06:01.745433 12692 tablet_service.cc:1467] Processing CreateTablet for tablet 31d74183a34b4ff2a6300a506140dbe6 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 02:06:01.746026 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ea486dc5bc4846b8e5317ea57ffbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.746354 12624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83394d7a6987493fbe49151753f5df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.750733 12617 tablet_service.cc:1467] Processing CreateTablet for tablet 31d74183a34b4ff2a6300a506140dbe6 (DEFAULT_TABLE table=test-workload [id=f21cc1b0b0ad47a98920fa57d07a69f1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 02:06:01.752866 12692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d74183a34b4ff2a6300a506140dbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.757978 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f402c817de4a92874891a7efaeb2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.760437 12694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e55ac373a747e6bb8a5ebbd7ca120f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.765034 12623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eabecfb83454bb2936a70288a4cb5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.777597 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eb4bfbb3a54b4aade2249ab5390b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.752350 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d74183a34b4ff2a6300a506140dbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.780481 12620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f402c817de4a92874891a7efaeb2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.782594 12617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d74183a34b4ff2a6300a506140dbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.790132 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eabecfb83454bb2936a70288a4cb5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.799976 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d69aa35cb34fa78e7f44ac709e64c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.798461 12621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d69aa35cb34fa78e7f44ac709e64c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.802028 12693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eb4bfbb3a54b4aade2249ab5390b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:01.815722 12619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e55ac373a747e6bb8a5ebbd7ca120f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:06:01.933981 12580 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:06:01.983986 12829 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:01.997269 12828 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.005353 12827 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.035434 12829 tablet_bootstrap.cc:654] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.050180 12828 tablet_bootstrap.cc:654] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.051357 12827 tablet_bootstrap.cc:654] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.058976 12829 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.059490 12829 ts_tablet_manager.cc:1367] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.011s
I20240503 02:06:02.062317 12829 raft_consensus.cc:348] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.063163 12829 raft_consensus.cc:374] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.063452 12829 raft_consensus.cc:729] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.081900 12829 consensus_queue.cc:260] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.083794 12828 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.084161 12827 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.084266 12828 ts_tablet_manager.cc:1367] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.106s user 0.011s sys 0.013s
I20240503 02:06:02.084723 12827 ts_tablet_manager.cc:1367] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.105s user 0.018s sys 0.004s
I20240503 02:06:02.087393 12828 raft_consensus.cc:348] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.088188 12828 raft_consensus.cc:374] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.087522 12827 raft_consensus.cc:348] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.088515 12828 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.088676 12827 raft_consensus.cc:374] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.089181 12827 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.089412 12828 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.090070 12827 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.093586 12660 heartbeater.cc:503] Master 127.12.53.62:42995 was elected leader, sending a full tablet report...
I20240503 02:06:02.094054 12829 ts_tablet_manager.cc:1398] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.034s user 0.014s sys 0.001s
I20240503 02:06:02.095094 12829 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.097435 12827 ts_tablet_manager.cc:1398] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20240503 02:06:02.098285 12735 heartbeater.cc:503] Master 127.12.53.62:42995 was elected leader, sending a full tablet report...
I20240503 02:06:02.098299 12827 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.102948 12809 heartbeater.cc:503] Master 127.12.53.62:42995 was elected leader, sending a full tablet report...
I20240503 02:06:02.103948 12828 ts_tablet_manager.cc:1398] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.019s user 0.001s sys 0.013s
I20240503 02:06:02.104358 12827 tablet_bootstrap.cc:654] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.104885 12828 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.114929 12829 tablet_bootstrap.cc:654] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.115468 12828 tablet_bootstrap.cc:654] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.120533 12827 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.121116 12827 ts_tablet_manager.cc:1367] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20240503 02:06:02.123510 12827 raft_consensus.cc:348] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.124352 12827 raft_consensus.cc:374] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.124835 12827 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.125695 12827 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.127367 12827 ts_tablet_manager.cc:1398] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240503 02:06:02.128625 12827 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.130393 12829 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.132145 12829 ts_tablet_manager.cc:1367] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.009s
I20240503 02:06:02.136569 12827 tablet_bootstrap.cc:654] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.131446 12828 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.137264 12828 ts_tablet_manager.cc:1367] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.008s
W20240503 02:06:02.138886 12661 tablet.cc:2369] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:06:02.141309 12828 raft_consensus.cc:348] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.142130 12828 raft_consensus.cc:374] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.142464 12828 raft_consensus.cc:729] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.143182 12828 consensus_queue.cc:260] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.145539 12828 ts_tablet_manager.cc:1398] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20240503 02:06:02.146412 12828 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.151880 12828 tablet_bootstrap.cc:654] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.155339 12827 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.155788 12827 ts_tablet_manager.cc:1367] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.003s
I20240503 02:06:02.158301 12827 raft_consensus.cc:348] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.159063 12827 raft_consensus.cc:374] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.159356 12827 raft_consensus.cc:729] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.137128 12829 raft_consensus.cc:348] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.160003 12827 consensus_queue.cc:260] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.159991 12829 raft_consensus.cc:374] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.162329 12829 raft_consensus.cc:729] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.162988 12829 consensus_queue.cc:260] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.163515 12827 ts_tablet_manager.cc:1398] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 02:06:02.164717 12827 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.168690 12828 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.169258 12828 ts_tablet_manager.cc:1367] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.009s
I20240503 02:06:02.170382 12827 tablet_bootstrap.cc:654] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.171231 12829 ts_tablet_manager.cc:1398] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20240503 02:06:02.171986 12828 raft_consensus.cc:348] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.172544 12829 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.172830 12828 raft_consensus.cc:374] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.173197 12828 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.173889 12828 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.176318 12828 ts_tablet_manager.cc:1398] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240503 02:06:02.177233 12828 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.183352 12827 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.183826 12827 ts_tablet_manager.cc:1367] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240503 02:06:02.185763 12828 tablet_bootstrap.cc:654] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.186367 12827 raft_consensus.cc:348] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.187086 12827 raft_consensus.cc:374] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.187384 12827 raft_consensus.cc:729] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.188037 12827 consensus_queue.cc:260] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.190125 12827 ts_tablet_manager.cc:1398] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240503 02:06:02.191028 12827 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.197125 12834 raft_consensus.cc:482] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.197450 12829 tablet_bootstrap.cc:654] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.197575 12834 raft_consensus.cc:504] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.200299 12834 leader_election.cc:288] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.201347 12828 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.201879 12828 ts_tablet_manager.cc:1367] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20240503 02:06:02.204159 12827 tablet_bootstrap.cc:654] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.204593 12828 raft_consensus.cc:348] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.205405 12828 raft_consensus.cc:374] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.205724 12828 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.206434 12828 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.214154 12834 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.214612 12834 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.215306 12829 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.215721 12829 ts_tablet_manager.cc:1367] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.004s
I20240503 02:06:02.216444 12834 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.218828 12829 raft_consensus.cc:348] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.219543 12829 raft_consensus.cc:374] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.219738 12834 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.219892 12829 raft_consensus.cc:729] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.220165 12834 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.220557 12829 consensus_queue.cc:260] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.222021 12834 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.211447 12828 ts_tablet_manager.cc:1398] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20240503 02:06:02.222759 12829 ts_tablet_manager.cc:1398] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:06:02.223475 12828 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.223582 12829 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.242689 12827 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.249289 12827 ts_tablet_manager.cc:1367] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.058s user 0.025s sys 0.018s
I20240503 02:06:02.251999 12829 tablet_bootstrap.cc:654] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.252089 12827 raft_consensus.cc:348] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.253022 12827 raft_consensus.cc:374] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.253332 12827 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.254598 12828 tablet_bootstrap.cc:654] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.255092 12827 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.259335 12835 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.259891 12835 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.264490 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.270560 12835 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457), 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:02.264278 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.271546 12835 raft_consensus.cc:482] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.272634 12835 raft_consensus.cc:504] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.262624 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.275029 12835 leader_election.cc:288] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.276851 12709 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 0.
W20240503 02:06:02.278283 12747 leader_election.cc:341] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.279726 12747 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:02.280599 12834 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:06:02.280697 12747 leader_election.cc:341] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.281091 12834 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.281510 12834 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.295627 12834 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.297593 12835 raft_consensus.cc:482] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.299072 12835 raft_consensus.cc:504] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.305099 12835 leader_election.cc:288] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.320014 12834 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.323839 12827 ts_tablet_manager.cc:1398] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.074s user 0.006s sys 0.005s
I20240503 02:06:02.323912 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:02.323827 12833 raft_consensus.cc:482] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.324941 12827 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.324690 12708 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.324826 12833 raft_consensus.cc:504] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.333488 12633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.332916 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.334335 12635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.335965 12834 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.341176 12708 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 1.
I20240503 02:06:02.337107 12835 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.333580 12833 leader_election.cc:288] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.341553 12834 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.343068 12835 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.345667 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.344983 12747 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:02.350737 12834 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.352465 12834 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.350750 12835 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.354454 12834 raft_consensus.cc:686] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:02.355340 12834 consensus_queue.cc:237] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.351673 12829 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.359251 12827 tablet_bootstrap.cc:654] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.361984 12829 ts_tablet_manager.cc:1367] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.139s user 0.014s sys 0.027s
I20240503 02:06:02.379999 12829 raft_consensus.cc:348] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.380842 12829 raft_consensus.cc:374] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.381156 12829 raft_consensus.cc:729] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.385012 12829 consensus_queue.cc:260] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
W20240503 02:06:02.396843 12747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.2:40495, user_credentials={real_user=slave}} blocked reactor thread for 51946.7us
I20240503 02:06:02.401755 12828 tablet_bootstrap.cc:492] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.403400 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
W20240503 02:06:02.402460 12749 leader_election.cc:341] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.404054 12708 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 0.
I20240503 02:06:02.404211 12749 leader_election.cc:302] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed; no voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:02.403213 12828 ts_tablet_manager.cc:1367] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.180s user 0.006s sys 0.018s
I20240503 02:06:02.404614 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.405900 12857 raft_consensus.cc:2726] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:06:02.406246 12749 leader_election.cc:341] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.406891 12829 ts_tablet_manager.cc:1398] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.040s user 0.005s sys 0.002s
I20240503 02:06:02.406661 12749 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed; no voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:02.409214 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.409816 12634 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.410801 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
W20240503 02:06:02.411212 12749 leader_election.cc:341] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.411381 12632 raft_consensus.cc:2454] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.411243 12827 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
W20240503 02:06:02.412830 12749 leader_election.cc:341] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.411615 12828 raft_consensus.cc:348] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.410300 12857 raft_consensus.cc:2726] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:02.413934 12828 raft_consensus.cc:374] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.408859 12747 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:02.420004 12833 raft_consensus.cc:482] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.418035 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.414537 12827 ts_tablet_manager.cc:1367] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.004s
I20240503 02:06:02.420997 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.414336 12828 raft_consensus.cc:729] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.416699 12829 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.414784 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
W20240503 02:06:02.423025 12675 leader_election.cc:341] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.413969 12527 catalog_manager.cc:5484] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 1 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:02.421104 12833 raft_consensus.cc:504] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.423355 12857 raft_consensus.cc:2781] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.429723 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.431067 12857 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.433262 12857 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.433218 12675 leader_election.cc:302] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:02.434389 12833 leader_election.cc:288] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.434829 12835 raft_consensus.cc:2781] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.435178 12835 raft_consensus.cc:482] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.435458 12835 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:06:02.433043 12749 leader_election.cc:341] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.438655 12675 leader_election.cc:302] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
W20240503 02:06:02.439878 12675 leader_election.cc:341] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.440395 12849 raft_consensus.cc:2781] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.432263 12828 consensus_queue.cc:260] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.440897 12849 raft_consensus.cc:482] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 02:06:02.443324 12676 leader_election.cc:341] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.439855 12782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.441628 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.442616 12783 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.451746 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.444232 12849 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.444658 12676 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988; no voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:02.453909 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.455976 12850 raft_consensus.cc:2726] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:02.456108 12827 raft_consensus.cc:348] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
W20240503 02:06:02.455214 12676 leader_election.cc:341] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.456893 12827 raft_consensus.cc:374] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.457221 12827 raft_consensus.cc:729] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
W20240503 02:06:02.456089 12598 leader_election.cc:341] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.458214 12827 consensus_queue.cc:260] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
W20240503 02:06:02.458719 12598 leader_election.cc:341] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.460633 12827 ts_tablet_manager.cc:1398] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.036s user 0.005s sys 0.004s
I20240503 02:06:02.455741 12828 ts_tablet_manager.cc:1398] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20240503 02:06:02.460984 12835 raft_consensus.cc:504] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.463912 12827 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.469455 12828 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.470984 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:02.471721 12783 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.472333 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.473367 12835 leader_election.cc:288] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.474756 12632 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.483974 12849 raft_consensus.cc:504] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.486866 12849 leader_election.cc:288] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.487479 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.488139 12829 tablet_bootstrap.cc:654] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.488062 12634 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.489338 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:02.489890 12783 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
W20240503 02:06:02.493511 12676 leader_election.cc:341] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.494236 12676 leader_election.cc:302] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:02.494546 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.494988 12849 raft_consensus.cc:2781] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.486685 12857 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.496368 12827 tablet_bootstrap.cc:654] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
W20240503 02:06:02.504716 12601 leader_election.cc:341] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.496330 12849 raft_consensus.cc:686] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:02.506559 12601 leader_election.cc:302] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d1bc4920bb47a08fb9e750fd9b489e; no voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:02.510042 12833 raft_consensus.cc:2726] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:02.510216 12828 tablet_bootstrap.cc:654] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.495992 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.512905 12782 raft_consensus.cc:2379] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cd93fb4924f4c19ac7114f63d136988 in current term 1: Already voted for candidate 8a4ef27ae015452da6e963fea8a35bed in this term.
I20240503 02:06:02.509999 12849 consensus_queue.cc:237] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.515503 12676 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988; no voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:02.516168 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:02.517043 12835 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.519282 12783 raft_consensus.cc:2454] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 0.
I20240503 02:06:02.519631 12633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.520478 12632 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.521485 12601 leader_election.cc:302] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters: 3cd93fb4924f4c19ac7114f63d136988
W20240503 02:06:02.522199 12749 leader_election.cc:341] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.523221 12833 raft_consensus.cc:2781] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.523617 12833 raft_consensus.cc:482] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.524016 12833 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.516641 12857 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:02.526347 12634 raft_consensus.cc:2454] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.527549 12675 leader_election.cc:302] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters: 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:02.528914 12850 raft_consensus.cc:2781] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.542791 12850 raft_consensus.cc:686] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:02.544701 12835 raft_consensus.cc:2726] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:02.546643 12829 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.547143 12829 ts_tablet_manager.cc:1367] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.132s user 0.015s sys 0.016s
I20240503 02:06:02.550272 12829 raft_consensus.cc:348] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.551136 12829 raft_consensus.cc:374] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.551448 12829 raft_consensus.cc:729] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.551779 12709 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 1.
I20240503 02:06:02.552111 12829 consensus_queue.cc:260] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.553727 12747 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters: 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:02.555035 12857 raft_consensus.cc:2781] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.555523 12857 raft_consensus.cc:686] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:02.558679 12857 consensus_queue.cc:237] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.562112 12829 ts_tablet_manager.cc:1398] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20240503 02:06:02.563094 12829 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.563359 12850 consensus_queue.cc:237] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.566818 12827 tablet_bootstrap.cc:492] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.567188 12827 ts_tablet_manager.cc:1367] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.106s user 0.018s sys 0.006s
I20240503 02:06:02.568702 12829 tablet_bootstrap.cc:654] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.569084 12827 raft_consensus.cc:348] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.569710 12827 raft_consensus.cc:374] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.569952 12827 raft_consensus.cc:729] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.570497 12827 consensus_queue.cc:260] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.570644 12833 raft_consensus.cc:504] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.572233 12827 ts_tablet_manager.cc:1398] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:06:02.573278 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:02.573398 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:02.573940 12709 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.573962 12783 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.580046 12783 raft_consensus.cc:2454] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 1.
I20240503 02:06:02.580046 12709 raft_consensus.cc:2454] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 1.
I20240503 02:06:02.581629 12598 leader_election.cc:302] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:02.582311 12827 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.582780 12868 raft_consensus.cc:2781] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.588321 12829 tablet_bootstrap.cc:492] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.588845 12829 ts_tablet_manager.cc:1367] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.000s
I20240503 02:06:02.591437 12829 raft_consensus.cc:348] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.594688 12829 raft_consensus.cc:374] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.595042 12829 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.595871 12829 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.600790 12827 tablet_bootstrap.cc:654] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.600841 12531 catalog_manager.cc:5484] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:02.604797 12868 raft_consensus.cc:686] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:02.606613 12868 consensus_queue.cc:237] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.596186 12833 leader_election.cc:288] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.623544 12828 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.625950 12828 ts_tablet_manager.cc:1367] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.019s
I20240503 02:06:02.650091 12829 ts_tablet_manager.cc:1398] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.061s user 0.016s sys 0.009s
I20240503 02:06:02.651108 12829 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.651649 12827 tablet_bootstrap.cc:492] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.652108 12827 ts_tablet_manager.cc:1367] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.070s user 0.025s sys 0.000s
I20240503 02:06:02.657269 12827 raft_consensus.cc:348] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.658414 12827 raft_consensus.cc:374] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.658725 12827 raft_consensus.cc:729] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.659407 12827 consensus_queue.cc:260] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.661909 12827 ts_tablet_manager.cc:1398] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 02:06:02.662776 12827 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: Bootstrap starting.
I20240503 02:06:02.663362 12828 raft_consensus.cc:348] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.662854 12531 catalog_manager.cc:5484] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:02.664340 12828 raft_consensus.cc:374] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.671082 12828 raft_consensus.cc:729] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.671937 12828 consensus_queue.cc:260] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.674599 12828 ts_tablet_manager.cc:1398] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.048s user 0.007s sys 0.016s
I20240503 02:06:02.675524 12828 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.668960 12827 tablet_bootstrap.cc:654] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.670020 12526 catalog_manager.cc:5484] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 1 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:02.684005 12828 tablet_bootstrap.cc:654] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.691896 12850 raft_consensus.cc:482] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.692384 12850 raft_consensus.cc:504] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.694397 12850 leader_election.cc:288] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.695250 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.695904 12634 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.696944 12828 tablet_bootstrap.cc:492] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.697475 12828 ts_tablet_manager.cc:1367] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240503 02:06:02.696993 12675 leader_election.cc:302] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:02.698339 12850 raft_consensus.cc:2781] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.698664 12850 raft_consensus.cc:482] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.698995 12850 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.700510 12829 tablet_bootstrap.cc:654] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.696415 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.701411 12783 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.705741 12827 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: No bootstrap required, opened a new log
I20240503 02:06:02.706221 12827 ts_tablet_manager.cc:1367] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.008s
I20240503 02:06:02.712831 12526 catalog_manager.cc:5484] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 0 to 1, leader changed from <none> to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 1 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:02.715101 12827 raft_consensus.cc:348] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.715752 12827 raft_consensus.cc:374] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.714287 12850 raft_consensus.cc:504] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.716014 12827 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.717375 12849 raft_consensus.cc:482] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.717809 12849 raft_consensus.cc:504] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.718358 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.719135 12634 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.719715 12849 leader_election.cc:288] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.721187 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.722220 12782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
W20240503 02:06:02.722385 12676 leader_election.cc:341] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:02.722759 12782 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.724064 12828 raft_consensus.cc:348] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.724787 12828 raft_consensus.cc:374] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.725056 12828 raft_consensus.cc:729] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.725698 12828 consensus_queue.cc:260] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.727911 12828 ts_tablet_manager.cc:1398] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20240503 02:06:02.717927 12827 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.728758 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:02.729187 12850 leader_election.cc:288] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.729430 12632 raft_consensus.cc:2454] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:02.730808 12675 leader_election.cc:302] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters: 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:02.731942 12849 raft_consensus.cc:2781] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:02.732298 12849 raft_consensus.cc:482] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:02.732611 12849 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.737375 12827 ts_tablet_manager.cc:1398] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20240503 02:06:02.741218 12634 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.741792 12828 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.748245 12675 leader_election.cc:302] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:02.749178 12850 raft_consensus.cc:2781] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.749630 12850 raft_consensus.cc:686] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:02.745204 12849 raft_consensus.cc:504] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.750394 12850 consensus_queue.cc:237] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.752135 12849 leader_election.cc:288] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.745924 12829 tablet_bootstrap.cc:492] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.752846 12829 ts_tablet_manager.cc:1367] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.102s user 0.028s sys 0.011s
I20240503 02:06:02.753633 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:02.754316 12632 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.755509 12829 raft_consensus.cc:348] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.756204 12829 raft_consensus.cc:374] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.754249 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:02.756526 12829 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.756698 12783 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.757216 12829 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.759701 12829 ts_tablet_manager.cc:1398] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:06:02.760594 12829 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.762989 12783 raft_consensus.cc:2454] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.764027 12676 leader_election.cc:302] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:02.764848 12849 raft_consensus.cc:2781] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.765297 12849 raft_consensus.cc:686] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:02.766266 12829 tablet_bootstrap.cc:654] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.766044 12849 consensus_queue.cc:237] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.775635 12632 raft_consensus.cc:2454] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.747304 12782 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:02.780125 12828 tablet_bootstrap.cc:654] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.784443 12632 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.796962 12526 catalog_manager.cc:5484] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:02.800040 12857 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:02.798871 12828 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.801234 12828 ts_tablet_manager.cc:1367] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.000s
I20240503 02:06:02.803978 12828 raft_consensus.cc:348] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.807164 12828 raft_consensus.cc:374] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.807502 12828 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.808245 12828 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.814210 12828 ts_tablet_manager.cc:1398] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240503 02:06:02.815325 12828 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: Bootstrap starting.
I20240503 02:06:02.820387 12828 tablet_bootstrap.cc:654] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.822176 12531 catalog_manager.cc:5484] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:02.833048 12829 tablet_bootstrap.cc:492] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.833436 12829 ts_tablet_manager.cc:1367] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.032s
I20240503 02:06:02.835549 12829 raft_consensus.cc:348] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.836280 12829 raft_consensus.cc:374] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.836515 12829 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.840034 12829 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:02.844674 12828 tablet_bootstrap.cc:492] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: No bootstrap required, opened a new log
I20240503 02:06:02.845142 12828 ts_tablet_manager.cc:1367] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.007s
I20240503 02:06:02.847421 12828 raft_consensus.cc:348] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.848147 12828 raft_consensus.cc:374] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.848405 12828 raft_consensus.cc:729] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.849017 12828 consensus_queue.cc:260] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.857086 12829 ts_tablet_manager.cc:1398] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20240503 02:06:02.864679 12829 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: Bootstrap starting.
I20240503 02:06:02.869820 12829 tablet_bootstrap.cc:654] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.875618 12828 ts_tablet_manager.cc:1398] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: Time spent starting tablet: real 0.030s user 0.015s sys 0.012s
I20240503 02:06:02.873284 12834 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:02.880734 12829 tablet_bootstrap.cc:492] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: No bootstrap required, opened a new log
I20240503 02:06:02.881300 12829 ts_tablet_manager.cc:1367] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240503 02:06:02.884558 12829 raft_consensus.cc:348] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.885538 12829 raft_consensus.cc:374] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.885864 12829 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.886533 12829 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.889493 12829 ts_tablet_manager.cc:1398] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:06:02.906839 12868 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:02.907229 12868 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:02.908664 12868 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:02.909796 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:02.909798 12782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:02.910869 12709 raft_consensus.cc:2379] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99d1bc4920bb47a08fb9e750fd9b489e in current term 1: Already voted for candidate 8a4ef27ae015452da6e963fea8a35bed in this term.
I20240503 02:06:02.912776 12598 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d1bc4920bb47a08fb9e750fd9b489e; no voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:02.913408 12868 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.917642 12868 raft_consensus.cc:2726] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:06:02.935971 12580 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:06:02.940912 12893 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:02.951074 12893 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:02.972410 12849 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:02.981612 12835 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.059907 12868 consensus_queue.cc:1035] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.074059 12875 consensus_queue.cc:1035] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:03.075898 12893 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:03.076367 12893 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.078281 12893 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:03.079591 12782 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.080444 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988" is_pre_election: true
I20240503 02:06:03.081115 12708 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 0.
I20240503 02:06:03.083601 12747 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:03.084918 12893 raft_consensus.cc:2781] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:03.085292 12893 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:03.085623 12893 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.087671 12850 consensus_queue.cc:1035] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.089515 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:03.090103 12634 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 0.
I20240503 02:06:03.100353 12849 consensus_queue.cc:1035] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.108098 12893 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.112681 12893 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:03.113847 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.114504 12709 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.115998 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:03.116535 12634 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.120424 12634 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 1.
I20240503 02:06:03.120929 12709 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 1.
I20240503 02:06:03.121315 12749 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.122115 12893 raft_consensus.cc:2781] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:03.122555 12893 raft_consensus.cc:686] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:03.123253 12893 consensus_queue.cc:237] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.129732 12531 catalog_manager.cc:5484] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 1 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.145359 12835 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:03.145885 12835 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.147835 12835 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.148417 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" is_pre_election: true
I20240503 02:06:03.149050 12634 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:03.149319 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed" is_pre_election: true
I20240503 02:06:03.149765 12675 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.149945 12783 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 0.
I20240503 02:06:03.150393 12835 raft_consensus.cc:2781] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:03.150738 12835 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:03.151000 12835 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.154996 12835 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.156509 12835 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.157378 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:03.157547 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.157979 12634 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.158100 12783 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:03.164608 12783 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:03.164053 12634 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 1.
I20240503 02:06:03.165694 12676 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:03.166610 12849 raft_consensus.cc:2781] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:03.166069 12835 consensus_queue.cc:1035] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:03.167069 12849 raft_consensus.cc:686] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.167870 12849 consensus_queue.cc:237] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.178684 12849 consensus_queue.cc:1035] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.178236 12529 catalog_manager.cc:5484] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.201035 12849 consensus_queue.cc:1035] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.210533 12835 consensus_queue.cc:1035] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.495191 12783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:42836
I20240503 02:06:03.496030 12783 raft_consensus.cc:595] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:03.502576 12783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:42836
I20240503 02:06:03.503196 12783 raft_consensus.cc:595] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:03.515841 12893 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.517838 12709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:35254
I20240503 02:06:03.518452 12709 raft_consensus.cc:595] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:03.527244 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:35254
I20240503 02:06:03.527751 12708 raft_consensus.cc:595] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:03.535347 12892 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.545351 12783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:42836
I20240503 02:06:03.545993 12783 raft_consensus.cc:595] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:03.560403 12857 raft_consensus.cc:979] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.560914 12893 raft_consensus.cc:1067] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.562695 12634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59160
I20240503 02:06:03.563311 12634 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.563663 12634 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.570343 12634 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.572356 12634 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Requested vote from peers 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:03.575217 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.575801 12783 raft_consensus.cc:3032] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.576100 12783 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:03.576031 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.576795 12709 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.577066 12783 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.578480 12783 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.576157 12632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:59186
I20240503 02:06:03.581574 12632 raft_consensus.cc:595] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:03.584638 12835 raft_consensus.cc:979] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.585078 12835 raft_consensus.cc:1067] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.586261 12709 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.587208 12783 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.587450 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:35254
I20240503 02:06:03.587450 12598 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.588070 12708 raft_consensus.cc:595] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Received request to transfer leadership to TS 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:03.588690 12634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59172
I20240503 02:06:03.588972 12875 raft_consensus.cc:2781] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.589258 12634 raft_consensus.cc:482] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.589646 12634 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.597463 12875 raft_consensus.cc:686] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:03.598285 12865 raft_consensus.cc:979] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:03.598968 12875 consensus_queue.cc:237] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:03.599031 12865 raft_consensus.cc:1067] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Signalling peer 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:03.601019 12783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:42822
I20240503 02:06:03.602123 12783 raft_consensus.cc:482] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.602483 12783 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.602546 12634 raft_consensus.cc:504] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.604905 12865 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.605145 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.605695 12708 raft_consensus.cc:3032] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.606003 12708 raft_consensus.cc:729] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.606338 12526 catalog_manager.cc:5484] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 1 to 2, leader changed from 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 2 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:03.606729 12708 consensus_queue.cc:260] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.607975 12708 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.615357 12708 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.617317 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.623204 12784 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.631625 12784 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.627115 12875 raft_consensus.cc:979] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:03.621943 12783 raft_consensus.cc:504] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.622676 12634 leader_election.cc:288] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.633039 12875 raft_consensus.cc:1067] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:03.635044 12708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35240
I20240503 02:06:03.635741 12708 raft_consensus.cc:482] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.635700 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.636281 12708 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.636219 12598 leader_election.cc:302] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.636669 12709 raft_consensus.cc:3032] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.637022 12709 raft_consensus.cc:729] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.637742 12709 consensus_queue.cc:260] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.638880 12709 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.640805 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d74183a34b4ff2a6300a506140dbe6" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:03.641384 12634 raft_consensus.cc:3037] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.647089 12918 raft_consensus.cc:2781] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.647578 12918 raft_consensus.cc:686] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:03.648394 12918 consensus_queue.cc:237] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.649389 12634 raft_consensus.cc:2454] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 2.
I20240503 02:06:03.634714 12783 leader_election.cc:288] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 2 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:03.652675 12850 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:03.654438 12749 leader_election.cc:302] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.654613 12708 raft_consensus.cc:504] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.655382 12915 raft_consensus.cc:2781] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.658188 12529 catalog_manager.cc:5484] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 1 to 2, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 2 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.657871 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.670650 12783 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.659338 12915 raft_consensus.cc:686] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 2 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:03.659359 12709 raft_consensus.cc:2454] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 2.
I20240503 02:06:03.657287 12708 leader_election.cc:288] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 2 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.677538 12915 consensus_queue.cc:237] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.665586 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eb4bfbb3a54b4aade2249ab5390b60" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:03.679078 12632 raft_consensus.cc:3032] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.679450 12632 raft_consensus.cc:729] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:03.680188 12632 consensus_queue.cc:260] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.681432 12632 raft_consensus.cc:3037] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.686478 12783 raft_consensus.cc:2454] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 2.
I20240503 02:06:03.688666 12676 leader_election.cc:302] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:03.688709 12850 raft_consensus.cc:979] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.687649 12632 raft_consensus.cc:2454] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 2.
I20240503 02:06:03.689962 12850 raft_consensus.cc:2781] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.690426 12850 raft_consensus.cc:686] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.690518 12835 raft_consensus.cc:1067] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.691206 12850 consensus_queue.cc:237] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.692018 12526 catalog_manager.cc:5484] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 1 to 2, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 2 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:03.696266 12632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59172
I20240503 02:06:03.696844 12632 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.697185 12632 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.703331 12531 catalog_manager.cc:5484] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 1 to 2, leader changed from 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 2 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.703979 12632 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.706323 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.706560 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.706943 12708 raft_consensus.cc:3032] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.707166 12783 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.707302 12708 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.708091 12708 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.709146 12708 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.713891 12783 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.705989 12632 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.715082 12601 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.716168 12918 raft_consensus.cc:2781] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.716579 12918 raft_consensus.cc:686] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:03.717371 12918 consensus_queue.cc:237] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.718276 12708 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.723330 12526 catalog_manager.cc:5484] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 1 to 2, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 2 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.871522 12857 raft_consensus.cc:979] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:03.871933 12915 raft_consensus.cc:1067] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:03.873270 12708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35232
I20240503 02:06:03.873734 12708 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.874020 12708 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.877369 12857 raft_consensus.cc:979] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.877933 12915 raft_consensus.cc:1067] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:03.879427 12632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59160
I20240503 02:06:03.879971 12632 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:03.879727 12708 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.880376 12632 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.881372 12708 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 2 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.882203 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:03.882550 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.882815 12634 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.883155 12783 raft_consensus.cc:3032] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.883458 12783 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:03.884112 12783 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.885033 12783 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.886385 12632 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.888587 12632 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:03.888912 12634 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 2.
I20240503 02:06:03.889523 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:03.890048 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:03.890103 12784 raft_consensus.cc:3032] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:03.890103 12783 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 2.
I20240503 02:06:03.890431 12675 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.890588 12708 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.890657 12784 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:03.891414 12850 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.891485 12784 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.891952 12850 raft_consensus.cc:686] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:03.892778 12784 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:03.892845 12850 consensus_queue.cc:237] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.896981 12708 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.898303 12598 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:03.899061 12918 raft_consensus.cc:2781] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:03.899533 12918 raft_consensus.cc:686] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:03.900144 12918 consensus_queue.cc:237] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:03.900467 12526 catalog_manager.cc:5484] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 1 to 2, leader changed from 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 2 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.902213 12784 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:03.906414 12531 catalog_manager.cc:5484] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 1 to 2, leader changed from 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 2 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:03.997928 12784 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:03.999177 12918 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.006105 12708 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.007946 12922 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.098069 12784 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.099181 12922 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.106812 12708 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.108016 12922 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.183637 12784 raft_consensus.cc:1261] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.184799 12850 consensus_queue.cc:1035] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.185837 12708 raft_consensus.cc:1261] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.187009 12915 consensus_queue.cc:1035] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.192411 12634 raft_consensus.cc:1261] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.193713 12835 consensus_queue.cc:1035] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.204257 12632 raft_consensus.cc:1261] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.207930 12915 consensus_queue.cc:1035] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.226729 12784 raft_consensus.cc:1261] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.227768 12922 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.233897 12708 raft_consensus.cc:1261] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.235344 12918 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.287521 12784 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.288637 12865 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.296406 12632 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.298138 12835 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.427922 12784 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.429114 12918 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:04.436625 12708 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:04.437901 12922 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:06.612207 12500 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240503 02:06:06.613016 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:06.628293 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:06.628814 12708 raft_consensus.cc:595] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Received request to transfer leadership to TS 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:06.638002 12632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
mode: GRACEFUL
new_leader_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:59194
I20240503 02:06:06.638516 12632 raft_consensus.cc:595] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Received request to transfer leadership to TS 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:06.639511 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:06:06.639926 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:06.714970 12835 raft_consensus.cc:979] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:06.715413 12941 raft_consensus.cc:1067] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Signalling peer 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:06.716873 12783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:42822
I20240503 02:06:06.717365 12783 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:06.717633 12783 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.721841 12783 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.723229 12783 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 3 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:06.723979 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:06.724172 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:06.724570 12632 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.724678 12708 raft_consensus.cc:3032] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:06.725047 12708 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:06.725761 12708 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.726915 12708 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.730799 12632 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 3.
I20240503 02:06:06.731971 12749 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:06.732636 12937 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:06.732573 12708 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 3.
I20240503 02:06:06.733901 12937 raft_consensus.cc:686] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:06.735574 12937 consensus_queue.cc:237] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.738345 12922 raft_consensus.cc:979] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: : Instructing follower 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:06.738729 12943 raft_consensus.cc:1067] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Signalling peer 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:06.740348 12783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:42828
I20240503 02:06:06.740810 12783 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:06.741093 12783 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.745144 12526 catalog_manager.cc:5484] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 2 to 3, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 3 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:06.746959 12783 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.748919 12783 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 3 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:06.749644 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:06.750149 12632 raft_consensus.cc:3032] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:06.750399 12632 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:06.750353 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:06.751035 12708 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.751184 12632 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.752251 12632 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:06.756762 12708 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 3.
I20240503 02:06:06.757640 12747 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:06.757994 12632 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 3.
I20240503 02:06:06.758286 12937 raft_consensus.cc:2781] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:06.758828 12937 raft_consensus.cc:686] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:06.759828 12937 consensus_queue.cc:237] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:06.766934 12526 catalog_manager.cc:5484] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 2 to 3, leader changed from 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1) to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 3 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:07.205947 12632 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:07.207068 12937 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:07.215843 12708 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:07.217868 12937 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:07.229197 12708 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:07.230248 12937 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:07.238224 12632 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:07.240134 12937 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:07.640642 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:07.647159 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:07.647681 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:08.700613 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:08.705850 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:08.706306 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:06:09.583585 12805 debug-util.cc:398] Leaking SignalData structure 0x7b0800174060 after lost signal to thread 12503
W20240503 02:06:09.584317 12805 debug-util.cc:398] Leaking SignalData structure 0x7b0800174740 after lost signal to thread 12566
W20240503 02:06:09.585054 12805 debug-util.cc:398] Leaking SignalData structure 0x7b08001c2fe0 after lost signal to thread 12659
W20240503 02:06:09.585793 12805 debug-util.cc:398] Leaking SignalData structure 0x7b08001bfb80 after lost signal to thread 12734
W20240503 02:06:09.586438 12805 debug-util.cc:398] Leaking SignalData structure 0x7b0800161220 after lost signal to thread 12808
I20240503 02:06:09.707701 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:09.714251 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:09.714672 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:10.715384 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:10.719946 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:10.720301 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:11.721133 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:11.728109 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:11.728602 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:12.729416 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:12.734158 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:12.734555 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:13.735357 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:13.740214 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:13.740598 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:14.741372 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:14.746228 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:14.746681 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:15.747483 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:15.751993 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:15.752324 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:16.773624 12500 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240503 02:06:16.777618 12632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:59186
I20240503 02:06:16.778267 12632 raft_consensus.cc:595] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:16.782397 12632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:59186
I20240503 02:06:16.782882 12632 raft_consensus.cc:595] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:16.786697 12784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:42836
I20240503 02:06:16.787104 12784 raft_consensus.cc:595] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:16.791216 12632 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:59186
I20240503 02:06:16.791675 12632 raft_consensus.cc:595] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:16.798214 12784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
mode: GRACEFUL
new_leader_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:42836
I20240503 02:06:16.798672 12784 raft_consensus.cc:595] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Received request to transfer leadership to TS 3cd93fb4924f4c19ac7114f63d136988
I20240503 02:06:16.862747 12969 raft_consensus.cc:979] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.863262 12964 raft_consensus.cc:1067] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.864683 12709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35240
I20240503 02:06:16.865159 12709 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:16.865517 12709 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.870610 12709 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:16.872232 12709 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Requested vote from peers 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457), 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:16.873095 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:16.873163 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:16.873857 12784 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.873925 12632 raft_consensus.cc:3032] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:16.874239 12632 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:16.874821 12632 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:16.875804 12632 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.878300 12784 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:16.879418 12676 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:16.880298 12967 raft_consensus.cc:2781] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:16.880599 12632 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:16.880666 12967 raft_consensus.cc:686] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:16.881764 12967 consensus_queue.cc:237] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:16.890861 12526 catalog_manager.cc:5484] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 2 to 3, leader changed from 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 3 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:16.912464 12975 raft_consensus.cc:979] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.913035 12962 raft_consensus.cc:1067] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.914280 12709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35232
I20240503 02:06:16.914696 12709 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:16.914948 12709 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:16.917754 12969 raft_consensus.cc:979] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.918138 12964 raft_consensus.cc:1067] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:16.919538 12708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35240
I20240503 02:06:16.920017 12708 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:16.920399 12708 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.920063 12709 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.922029 12709 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 4 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:16.922524 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:16.923027 12632 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:16.925130 12708 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.926473 12708 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:16.926745 12632 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 4.
I20240503 02:06:16.927551 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:16.927693 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:16.927738 12675 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:16.928094 12784 raft_consensus.cc:3032] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:16.928232 12634 raft_consensus.cc:3032] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:16.928414 12784 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:16.928576 12967 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:16.928602 12634 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:16.929212 12967 raft_consensus.cc:686] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:16.929143 12784 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.929478 12634 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.930222 12784 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:16.930076 12967 consensus_queue.cc:237] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.930692 12634 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.934592 12784 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 4.
I20240503 02:06:16.935770 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:16.936275 12783 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:16.938086 12634 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:16.938956 12675 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:16.938588 12526 catalog_manager.cc:5484] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 3 to 4, leader changed from 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 4 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:16.939574 12967 raft_consensus.cc:2781] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:16.939962 12967 raft_consensus.cc:686] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:16.940634 12967 consensus_queue.cc:237] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:16.941666 12783 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:16.949973 12526 catalog_manager.cc:5484] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 2 to 3, leader changed from 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 3 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.155730 12969 raft_consensus.cc:979] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:17.156108 12964 raft_consensus.cc:1067] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:17.157404 12708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35240
I20240503 02:06:17.157837 12708 raft_consensus.cc:482] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:17.158104 12708 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:17.161975 12708 raft_consensus.cc:504] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.163200 12708 leader_election.cc:288] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:17.163971 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:17.164144 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ea486dc5bc4846b8e5317ea57ffbc8" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:17.164414 12634 raft_consensus.cc:3032] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:17.164644 12783 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:17.164693 12634 raft_consensus.cc:729] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:17.165452 12634 consensus_queue.cc:260] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.166319 12634 raft_consensus.cc:3037] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:17.168520 12783 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:17.169359 12676 leader_election.cc:302] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:17.169986 12967 raft_consensus.cc:2781] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:17.170449 12967 raft_consensus.cc:686] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:17.170475 12634 raft_consensus.cc:2454] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 3.
I20240503 02:06:17.171365 12967 consensus_queue.cc:237] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.177238 12526 catalog_manager.cc:5484] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 2 to 3, leader changed from 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 3 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.263360 12975 raft_consensus.cc:979] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: : Instructing follower 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:17.263751 12962 raft_consensus.cc:1067] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Signalling peer 3cd93fb4924f4c19ac7114f63d136988 to start an election
I20240503 02:06:17.265051 12708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
from {username='slave'} at 127.0.0.1:35232
I20240503 02:06:17.265472 12708 raft_consensus.cc:482] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:17.265718 12708 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:17.269589 12708 raft_consensus.cc:504] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.271378 12708 leader_election.cc:288] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 4 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:17.272158 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:17.272274 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eabecfb83454bb2936a70288a4cb5bf" candidate_uuid: "3cd93fb4924f4c19ac7114f63d136988" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:17.272601 12634 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:17.272768 12783 raft_consensus.cc:3032] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:17.273061 12783 raft_consensus.cc:729] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:17.273547 12783 consensus_queue.cc:260] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.274360 12783 raft_consensus.cc:3037] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:17.276939 12784 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.277027 12634 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 4.
I20240503 02:06:17.278021 12675 leader_election.cc:302] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:17.278164 12967 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:17.278914 12941 raft_consensus.cc:2781] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:17.279112 12783 raft_consensus.cc:2454] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd93fb4924f4c19ac7114f63d136988 in term 4.
I20240503 02:06:17.279806 12941 raft_consensus.cc:686] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Becoming Leader. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:17.280622 12941 consensus_queue.cc:237] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:17.290999 12634 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.293138 12967 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:17.295325 12526 catalog_manager.cc:5484] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 reported cstate change: term changed from 3 to 4, leader changed from 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3) to 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2). New cstate: current_term: 4 leader_uuid: "3cd93fb4924f4c19ac7114f63d136988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.386023 12634 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.387089 12967 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:17.395853 12783 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.397455 12981 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:06:17.473155 12634 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:17.474303 12981 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:17.475050 12783 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:17.476158 12981 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:17.667620 12634 raft_consensus.cc:1261] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.668604 12941 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:17.677445 12783 raft_consensus.cc:1261] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:17.678591 12981 consensus_queue.cc:1035] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:17.688169 12783 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:17.689208 12941 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:17.699687 12634 raft_consensus.cc:1261] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Refusing update from remote peer 3cd93fb4924f4c19ac7114f63d136988: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:17.701154 12981 consensus_queue.cc:1035] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:19.829824 12500 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240503 02:06:19.830590 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:19.837821 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:19.838445 12708 raft_consensus.cc:595] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:19.840603 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:19.841069 12708 raft_consensus.cc:595] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Received request to transfer leadership to TS 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:19.842902 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:19.843349 12708 raft_consensus.cc:595] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:19.845081 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:19.845460 12708 raft_consensus.cc:595] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Received request to transfer leadership to TS 8a4ef27ae015452da6e963fea8a35bed
I20240503 02:06:19.847132 12708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
mode: GRACEFUL
new_leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:35260
I20240503 02:06:19.847501 12708 raft_consensus.cc:595] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Received request to transfer leadership to TS 99d1bc4920bb47a08fb9e750fd9b489e
I20240503 02:06:19.848258 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240503 02:06:19.848654 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:19.970072 12981 raft_consensus.cc:979] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:19.970521 12981 raft_consensus.cc:1067] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:19.971935 12632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83394d7a6987493fbe49151753f5df90"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59172
I20240503 02:06:19.972368 12632 raft_consensus.cc:482] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:19.972720 12632 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:19.979326 12632 raft_consensus.cc:504] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:19.981747 12632 leader_election.cc:288] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 4 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:19.982913 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:19.982762 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83394d7a6987493fbe49151753f5df90" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:19.983671 12708 raft_consensus.cc:3032] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:19.983891 12708 raft_consensus.cc:729] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:19.984493 12708 consensus_queue.cc:260] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:19.984917 12783 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:19.985594 12708 raft_consensus.cc:3037] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:19.990413 12783 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 4.
I20240503 02:06:19.991789 12601 leader_election.cc:302] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:19.991883 12708 raft_consensus.cc:2454] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 4.
I20240503 02:06:19.993539 12996 raft_consensus.cc:2781] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:19.994589 12996 raft_consensus.cc:686] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:19.995694 12996 consensus_queue.cc:237] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.003836 12526 catalog_manager.cc:5484] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 3 to 4, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 4 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:20.055831 12995 raft_consensus.cc:979] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:20.056214 12981 raft_consensus.cc:1067] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Signalling peer 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:20.057543 12783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:42822
I20240503 02:06:20.057958 12783 raft_consensus.cc:482] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:20.058192 12783 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:20.062325 12783 raft_consensus.cc:504] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:20.064237 12783 leader_election.cc:288] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 4 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119)
I20240503 02:06:20.065048 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:20.065186 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afaeb28c1164547bf2677d565338ca5" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:20.065629 12708 raft_consensus.cc:3032] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:20.065637 12632 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:20.066047 12708 raft_consensus.cc:729] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:20.066752 12708 consensus_queue.cc:260] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:20.067899 12708 raft_consensus.cc:3037] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:20.071568 12632 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 4.
I20240503 02:06:20.072786 12749 leader_election.cc:302] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:20.073628 12962 raft_consensus.cc:2781] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:20.073685 12708 raft_consensus.cc:2454] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 4.
I20240503 02:06:20.074757 12962 raft_consensus.cc:686] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 4 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:20.075565 12962 consensus_queue.cc:237] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } }
I20240503 02:06:20.081678 12526 catalog_manager.cc:5484] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 3 to 4, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 4 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:20.218997 12995 raft_consensus.cc:979] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:20.219408 12981 raft_consensus.cc:1067] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Signalling peer 8a4ef27ae015452da6e963fea8a35bed to start an election
I20240503 02:06:20.220729 12783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f"
dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
from {username='slave'} at 127.0.0.1:42822
I20240503 02:06:20.221165 12783 raft_consensus.cc:482] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:20.221383 12783 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.226097 12995 raft_consensus.cc:979] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:20.226181 12783 raft_consensus.cc:504] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.226568 12981 raft_consensus.cc:1067] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:20.227624 12783 leader_election.cc:288] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 2 election: Requested vote from peers 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119), 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495)
I20240503 02:06:20.227859 12632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59172
I20240503 02:06:20.228365 12632 raft_consensus.cc:482] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:20.228701 12632 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.228541 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:20.228541 12634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e55ac373a747e6bb8a5ebbd7ca120f" candidate_uuid: "8a4ef27ae015452da6e963fea8a35bed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
I20240503 02:06:20.229324 12708 raft_consensus.cc:3032] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:20.229383 12634 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.229558 12708 raft_consensus.cc:729] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:20.230084 12708 consensus_queue.cc:260] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.230898 12708 raft_consensus.cc:3037] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.234787 12708 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 2.
I20240503 02:06:20.234594 12632 raft_consensus.cc:504] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.235386 12634 raft_consensus.cc:2454] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ef27ae015452da6e963fea8a35bed in term 2.
I20240503 02:06:20.235723 12747 leader_election.cc:302] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 8a4ef27ae015452da6e963fea8a35bed; no voters:
I20240503 02:06:20.236454 12962 raft_consensus.cc:2781] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:20.237290 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:20.236928 12962 raft_consensus.cc:686] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 2 LEADER]: Becoming Leader. State: Replica: 8a4ef27ae015452da6e963fea8a35bed, State: Running, Role: LEADER
I20240503 02:06:20.237749 12708 raft_consensus.cc:3032] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:20.238055 12708 raft_consensus.cc:729] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:20.238368 12962 consensus_queue.cc:237] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.238579 12708 consensus_queue.cc:260] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.239537 12708 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.244535 12708 raft_consensus.cc:2454] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:20.245507 12632 leader_election.cc:288] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:20.245515 12598 leader_election.cc:302] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd93fb4924f4c19ac7114f63d136988, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:20.246032 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d69aa35cb34fa78e7f44ac709e64c4" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:20.246423 12996 raft_consensus.cc:2781] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:20.246558 12783 raft_consensus.cc:3037] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.246912 12996 raft_consensus.cc:686] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:20.246876 12526 catalog_manager.cc:5484] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed reported cstate change: term changed from 1 to 2, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3). New cstate: current_term: 2 leader_uuid: "8a4ef27ae015452da6e963fea8a35bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:20.248101 12996 consensus_queue.cc:237] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.252486 12783 raft_consensus.cc:2454] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 2.
I20240503 02:06:20.257671 12526 catalog_manager.cc:5484] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 1 to 2, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 2 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:20.365464 12995 raft_consensus.cc:979] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: : Instructing follower 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:20.365854 12981 raft_consensus.cc:1067] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Signalling peer 99d1bc4920bb47a08fb9e750fd9b489e to start an election
I20240503 02:06:20.367089 12632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf"
dest_uuid: "99d1bc4920bb47a08fb9e750fd9b489e"
from {username='slave'} at 127.0.0.1:59172
I20240503 02:06:20.367486 12632 raft_consensus.cc:482] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:20.367739 12632 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:20.371989 12632 raft_consensus.cc:504] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.373417 12632 leader_election.cc:288] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 5 election: Requested vote from peers 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495), 8a4ef27ae015452da6e963fea8a35bed (127.12.53.3:46457)
I20240503 02:06:20.374200 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3cd93fb4924f4c19ac7114f63d136988"
I20240503 02:06:20.374320 12783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f402c817de4a92874891a7efaeb2cf" candidate_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8a4ef27ae015452da6e963fea8a35bed"
I20240503 02:06:20.374681 12708 raft_consensus.cc:3032] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:06:20.374732 12783 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:20.374959 12708 raft_consensus.cc:729] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3cd93fb4924f4c19ac7114f63d136988, State: Running, Role: LEADER
I20240503 02:06:20.375514 12708 consensus_queue.cc:260] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.376343 12708 raft_consensus.cc:3037] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:20.378646 12783 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 5.
I20240503 02:06:20.379398 12601 leader_election.cc:302] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ef27ae015452da6e963fea8a35bed, 99d1bc4920bb47a08fb9e750fd9b489e; no voters:
I20240503 02:06:20.379979 12996 raft_consensus.cc:2781] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:06:20.380303 12708 raft_consensus.cc:2454] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d1bc4920bb47a08fb9e750fd9b489e in term 5.
I20240503 02:06:20.380432 12996 raft_consensus.cc:686] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 5 LEADER]: Becoming Leader. State: Replica: 99d1bc4920bb47a08fb9e750fd9b489e, State: Running, Role: LEADER
I20240503 02:06:20.381237 12996 consensus_queue.cc:237] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } }
I20240503 02:06:20.387495 12526 catalog_manager.cc:5484] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e reported cstate change: term changed from 4 to 5, leader changed from 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2) to 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1). New cstate: current_term: 5 leader_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:20.418430 12708 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:20.419431 12996 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:20.429284 12783 raft_consensus.cc:1261] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:20.430593 12996 consensus_queue.cc:1035] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:20.679150 12632 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:20.680375 12962 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:20.688463 12708 raft_consensus.cc:1261] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:20.690507 13003 consensus_queue.cc:1035] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 02:06:20.699540 12632 raft_consensus.cc:1261] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:20.700655 13003 consensus_queue.cc:1035] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d1bc4920bb47a08fb9e750fd9b489e" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:20.710944 12708 raft_consensus.cc:1261] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 8a4ef27ae015452da6e963fea8a35bed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:20.712205 13003 consensus_queue.cc:1035] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:20.772210 12783 raft_consensus.cc:1261] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:20.773383 12996 consensus_queue.cc:1035] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:20.781780 12708 raft_consensus.cc:1261] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:20.783548 13020 consensus_queue.cc:1035] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:20.849474 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:20.854223 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:20.854616 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:20.878685 12708 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 5 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 02:06:20.879824 13020 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd93fb4924f4c19ac7114f63d136988" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:06:20.887224 12783 raft_consensus.cc:1261] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 5 FOLLOWER]: Refusing update from remote peer 99d1bc4920bb47a08fb9e750fd9b489e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 02:06:20.888373 13010 consensus_queue.cc:1035] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4ef27ae015452da6e963fea8a35bed" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:06:21.855388 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:21.860134 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:21.860528 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:22.861279 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:22.865952 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:22.866349 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:23.867166 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:23.873169 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:23.873556 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:24.874346 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:24.879436 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:24.879834 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:25.880676 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:25.885627 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:25.886008 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:26.886755 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:26.891389 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:26.891768 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:27.892534 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:27.897429 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:27.897822 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:28.898556 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:28.903244 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:28.903710 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:29.926493 12500 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240503 02:06:29.927132 12500 tablet_server.cc:178] TabletServer@127.12.53.1:0 shutting down...
W20240503 02:06:29.942481 12749 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:06:29.946072 12749 consensus_peers.cc:485] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed -> Peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Couldn't send request to peer 99d1bc4920bb47a08fb9e750fd9b489e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:29.959769 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:29.960467 12500 tablet_replica.cc:330] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.961159 12500 raft_consensus.cc:2229] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 5 LEADER]: Raft consensus shutting down.
I20240503 02:06:29.962080 12500 raft_consensus.cc:2258] T 05f402c817de4a92874891a7efaeb2cf P 99d1bc4920bb47a08fb9e750fd9b489e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.964352 12500 tablet_replica.cc:330] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.964892 12500 raft_consensus.cc:2229] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:06:29.965689 12500 raft_consensus.cc:2258] T 83394d7a6987493fbe49151753f5df90 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.967272 12500 tablet_replica.cc:330] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.967757 12500 raft_consensus.cc:2229] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.968179 12500 raft_consensus.cc:2258] T 68e55ac373a747e6bb8a5ebbd7ca120f P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.969734 12500 tablet_replica.cc:330] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.970182 12500 raft_consensus.cc:2229] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.970585 12500 raft_consensus.cc:2258] T 31d74183a34b4ff2a6300a506140dbe6 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.972054 12500 tablet_replica.cc:330] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.972488 12500 raft_consensus.cc:2229] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.972970 12500 raft_consensus.cc:2258] T 0afaeb28c1164547bf2677d565338ca5 P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.974435 12500 tablet_replica.cc:330] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.974864 12500 raft_consensus.cc:2229] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.975296 12500 raft_consensus.cc:2258] T e3eb4bfbb3a54b4aade2249ab5390b60 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.976730 12500 tablet_replica.cc:330] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.977205 12500 raft_consensus.cc:2229] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:29.977981 12500 raft_consensus.cc:2258] T 33d69aa35cb34fa78e7f44ac709e64c4 P 99d1bc4920bb47a08fb9e750fd9b489e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.979408 12500 tablet_replica.cc:330] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.979830 12500 raft_consensus.cc:2229] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.980259 12500 raft_consensus.cc:2258] T 4eabecfb83454bb2936a70288a4cb5bf P 99d1bc4920bb47a08fb9e750fd9b489e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:29.981905 12500 tablet_replica.cc:330] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e: stopping tablet replica
I20240503 02:06:29.982352 12500 raft_consensus.cc:2229] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:29.982867 12500 raft_consensus.cc:2258] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 99d1bc4920bb47a08fb9e750fd9b489e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.079342 12500 tablet_server.cc:194] TabletServer@127.12.53.1:0 shutdown complete.
W20240503 02:06:30.079825 12675 consensus_peers.cc:485] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 -> Peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Couldn't send request to peer 99d1bc4920bb47a08fb9e750fd9b489e. Status: Network error: Client connection negotiation failed: client connection to 127.12.53.1:40119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:06:30.080377 12675 consensus_peers.cc:485] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 -> Peer 99d1bc4920bb47a08fb9e750fd9b489e (127.12.53.1:40119): Couldn't send request to peer 99d1bc4920bb47a08fb9e750fd9b489e. Status: Network error: Client connection negotiation failed: client connection to 127.12.53.1:40119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:30.098668 12500 tablet_server.cc:178] TabletServer@127.12.53.2:0 shutting down...
W20240503 02:06:30.104223 12747 consensus_peers.cc:485] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed -> Peer 3cd93fb4924f4c19ac7114f63d136988 (127.12.53.2:40495): Couldn't send request to peer 3cd93fb4924f4c19ac7114f63d136988. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:30.122241 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:30.122834 12500 tablet_replica.cc:330] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.123486 12500 raft_consensus.cc:2229] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.123957 12500 raft_consensus.cc:2258] T 68e55ac373a747e6bb8a5ebbd7ca120f P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.125660 12500 tablet_replica.cc:330] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.126118 12500 raft_consensus.cc:2229] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.126523 12500 raft_consensus.cc:2258] T 31d74183a34b4ff2a6300a506140dbe6 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.128060 12500 tablet_replica.cc:330] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.128532 12500 raft_consensus.cc:2229] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.128966 12500 raft_consensus.cc:2258] T 05f402c817de4a92874891a7efaeb2cf P 3cd93fb4924f4c19ac7114f63d136988 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.130421 12500 tablet_replica.cc:330] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.130858 12500 raft_consensus.cc:2229] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.131592 12500 raft_consensus.cc:2258] T 4eabecfb83454bb2936a70288a4cb5bf P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.133098 12500 tablet_replica.cc:330] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.133549 12500 raft_consensus.cc:2229] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.134240 12500 raft_consensus.cc:2258] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 3cd93fb4924f4c19ac7114f63d136988 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.135715 12500 tablet_replica.cc:330] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.136134 12500 raft_consensus.cc:2229] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.136569 12500 raft_consensus.cc:2258] T 33d69aa35cb34fa78e7f44ac709e64c4 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.138262 12500 tablet_replica.cc:330] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.138676 12500 raft_consensus.cc:2229] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.139364 12500 raft_consensus.cc:2258] T e3eb4bfbb3a54b4aade2249ab5390b60 P 3cd93fb4924f4c19ac7114f63d136988 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.140810 12500 tablet_replica.cc:330] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.141240 12500 raft_consensus.cc:2229] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.141654 12500 raft_consensus.cc:2258] T 0afaeb28c1164547bf2677d565338ca5 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.143219 12500 tablet_replica.cc:330] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988: stopping tablet replica
I20240503 02:06:30.143628 12500 raft_consensus.cc:2229] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.144008 12500 raft_consensus.cc:2258] T 83394d7a6987493fbe49151753f5df90 P 3cd93fb4924f4c19ac7114f63d136988 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.170693 12500 tablet_server.cc:194] TabletServer@127.12.53.2:0 shutdown complete.
I20240503 02:06:30.188366 12500 tablet_server.cc:178] TabletServer@127.12.53.3:0 shutting down...
I20240503 02:06:30.210242 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:30.210891 12500 tablet_replica.cc:330] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.211572 12500 raft_consensus.cc:2229] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.212021 12500 raft_consensus.cc:2258] T 33d69aa35cb34fa78e7f44ac709e64c4 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.213832 12500 tablet_replica.cc:330] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.214299 12500 raft_consensus.cc:2229] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.215148 12500 raft_consensus.cc:2258] T 31d74183a34b4ff2a6300a506140dbe6 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.216773 12500 tablet_replica.cc:330] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.217248 12500 raft_consensus.cc:2229] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.217947 12500 raft_consensus.cc:2258] T 68e55ac373a747e6bb8a5ebbd7ca120f P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.219381 12500 tablet_replica.cc:330] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.219820 12500 raft_consensus.cc:2229] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.220260 12500 raft_consensus.cc:2258] T 83394d7a6987493fbe49151753f5df90 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.221825 12500 tablet_replica.cc:330] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.222272 12500 raft_consensus.cc:2229] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.222666 12500 raft_consensus.cc:2258] T f7ea486dc5bc4846b8e5317ea57ffbc8 P 8a4ef27ae015452da6e963fea8a35bed [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.224069 12500 tablet_replica.cc:330] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.224499 12500 raft_consensus.cc:2229] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.224916 12500 raft_consensus.cc:2258] T 4eabecfb83454bb2936a70288a4cb5bf P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.226351 12500 tablet_replica.cc:330] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.226764 12500 raft_consensus.cc:2229] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.227470 12500 raft_consensus.cc:2258] T 0afaeb28c1164547bf2677d565338ca5 P 8a4ef27ae015452da6e963fea8a35bed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.228922 12500 tablet_replica.cc:330] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.229342 12500 raft_consensus.cc:2229] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.229686 12500 raft_consensus.cc:2258] T e3eb4bfbb3a54b4aade2249ab5390b60 P 8a4ef27ae015452da6e963fea8a35bed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.231195 12500 tablet_replica.cc:330] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed: stopping tablet replica
I20240503 02:06:30.231602 12500 raft_consensus.cc:2229] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:30.231976 12500 raft_consensus.cc:2258] T 05f402c817de4a92874891a7efaeb2cf P 8a4ef27ae015452da6e963fea8a35bed [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.273798 12500 tablet_server.cc:194] TabletServer@127.12.53.3:0 shutdown complete.
I20240503 02:06:30.289736 12500 master.cc:537] Master@127.12.53.62:42995 shutting down...
I20240503 02:06:30.391498 12500 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:30.392079 12500 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:30.392357 12500 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66f348d5cb6048c5b3f7652e70c4076b: stopping tablet replica
I20240503 02:06:30.410115 12500 master.cc:552] Master@127.12.53.62:42995 shutdown complete.
3cd93fb4924f4c19ac7114f63d136988 4
8a4ef27ae015452da6e963fea8a35bed 1
99d1bc4920bb47a08fb9e750fd9b489e 4
3cd93fb4924f4c19ac7114f63d136988 3
8a4ef27ae015452da6e963fea8a35bed 3
99d1bc4920bb47a08fb9e750fd9b489e 3
3cd93fb4924f4c19ac7114f63d136988 8
8a4ef27ae015452da6e963fea8a35bed 1
99d1bc4920bb47a08fb9e750fd9b489e 0
3cd93fb4924f4c19ac7114f63d136988 3
8a4ef27ae015452da6e963fea8a35bed 3
99d1bc4920bb47a08fb9e750fd9b489e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30863 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240503 02:06:30.446938 12500 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.53.62:40893
I20240503 02:06:30.448004 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:30.453207 13034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.453974 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.454716 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:30.455639 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:30.456470 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:30.456661 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:30.456836 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701990456817 us; error 0 us; skew 500 ppm
I20240503 02:06:30.457321 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:30.459596 12500 webserver.cc:460] Webserver started at http://127.12.53.62:42485/ using document root <none> and password file <none>
I20240503 02:06:30.460047 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:30.460208 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:30.460484 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:30.461647 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/master-0-root/instance:
uuid: "ef0aa78c99c44b559a6cb65523d50423"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.465993 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:06:30.469060 13042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.469713 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:30.469975 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/master-0-root
uuid: "ef0aa78c99c44b559a6cb65523d50423"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.470242 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:30.482482 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:30.483661 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:30.518559 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.62:40893
I20240503 02:06:30.518647 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.62:40893 every 8 connection(s)
I20240503 02:06:30.522181 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:30.531105 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423: Bootstrap starting.
I20240503 02:06:30.535107 13094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:30.538924 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423: No bootstrap required, opened a new log
I20240503 02:06:30.540807 13094 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER }
I20240503 02:06:30.541244 13094 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:30.541420 13094 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0aa78c99c44b559a6cb65523d50423, State: Initialized, Role: FOLLOWER
I20240503 02:06:30.541853 13094 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER }
I20240503 02:06:30.542250 13094 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:06:30.542433 13094 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:06:30.542658 13094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:30.546350 13094 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER }
I20240503 02:06:30.546808 13094 leader_election.cc:302] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef0aa78c99c44b559a6cb65523d50423; no voters:
I20240503 02:06:30.547765 13094 leader_election.cc:288] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:06:30.548118 13097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:30.549374 13097 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 1 LEADER]: Becoming Leader. State: Replica: ef0aa78c99c44b559a6cb65523d50423, State: Running, Role: LEADER
I20240503 02:06:30.549952 13097 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER }
I20240503 02:06:30.550480 13094 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:06:30.554589 13099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0aa78c99c44b559a6cb65523d50423. Latest consensus state: current_term: 1 leader_uuid: "ef0aa78c99c44b559a6cb65523d50423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER } }
I20240503 02:06:30.554832 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef0aa78c99c44b559a6cb65523d50423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0aa78c99c44b559a6cb65523d50423" member_type: VOTER } }
I20240503 02:06:30.555189 13099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:30.555436 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:30.558310 13104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:06:30.563711 13104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:06:30.565647 12500 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:06:30.572134 13104 catalog_manager.cc:1347] Generated new cluster ID: 39eb567c46984fe0bbf7aee9f245b9fd
I20240503 02:06:30.572428 13104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:06:30.588647 13104 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:06:30.589856 13104 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:06:30.610491 13104 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423: Generated new TSK 0
I20240503 02:06:30.611074 13104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:06:30.631696 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:30.637367 13115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.638316 13116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:30.639783 12500 server_base.cc:1034] running on GCE node
W20240503 02:06:30.640197 13118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:30.640997 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:30.641196 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:30.641348 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701990641332 us; error 0 us; skew 500 ppm
I20240503 02:06:30.641865 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:30.644029 12500 webserver.cc:460] Webserver started at http://127.12.53.1:42261/ using document root <none> and password file <none>
I20240503 02:06:30.644495 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:30.644660 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:30.644929 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:30.646016 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-0-root/instance:
uuid: "d6ed2caa226f4e12ac39799a6e80e878"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.650221 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20240503 02:06:30.653311 13123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.653996 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:06:30.654263 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
uuid: "d6ed2caa226f4e12ac39799a6e80e878"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.654518 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:30.675251 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:30.676445 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:30.677865 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:30.679986 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:30.680178 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.680408 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:30.680564 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.717360 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.1:43915
I20240503 02:06:30.717468 13185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.1:43915 every 8 connection(s)
I20240503 02:06:30.721560 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:30.729538 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.729573 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.731935 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:30.733454 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:30.734807 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:06:30.734889 13186 heartbeater.cc:347] Connected to a master server at 127.12.53.62:40893
W20240503 02:06:30.735101 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:30.735270 13186 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:30.735304 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701990735285 us; error 0 us; skew 500 ppm
I20240503 02:06:30.736220 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:30.736227 13186 heartbeater.cc:511] Master 127.12.53.62:40893 requested a full tablet report, sending...
I20240503 02:06:30.738345 13059 ts_manager.cc:194] Registered new tserver with Master: d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915)
I20240503 02:06:30.739079 12500 webserver.cc:460] Webserver started at http://127.12.53.2:45277/ using document root <none> and password file <none>
I20240503 02:06:30.739554 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:30.739723 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:30.739964 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:30.740147 13059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58148
I20240503 02:06:30.741111 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-1-root/instance:
uuid: "ef52048cb739452e9b4b2266943ec12f"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.745564 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:06:30.748512 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.749231 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:30.749495 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
uuid: "ef52048cb739452e9b4b2266943ec12f"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.749754 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:30.765781 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:30.766919 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:30.768239 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:30.770360 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:30.770555 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.770785 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:30.770961 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.807396 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.2:38099
I20240503 02:06:30.807503 13260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.2:38099 every 8 connection(s)
I20240503 02:06:30.811781 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:30.818186 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.819221 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:30.821158 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:30.822340 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:30.823151 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:30.823383 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:30.823525 13261 heartbeater.cc:347] Connected to a master server at 127.12.53.62:40893
I20240503 02:06:30.823560 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714701990823542 us; error 0 us; skew 500 ppm
I20240503 02:06:30.823982 13261 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:30.824319 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:30.824651 13261 heartbeater.cc:511] Master 127.12.53.62:40893 requested a full tablet report, sending...
I20240503 02:06:30.826764 13059 ts_manager.cc:194] Registered new tserver with Master: ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:30.826967 12500 webserver.cc:460] Webserver started at http://127.12.53.3:36967/ using document root <none> and password file <none>
I20240503 02:06:30.827603 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:30.827836 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:30.828156 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:30.828495 13059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58150
I20240503 02:06:30.829689 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-2-root/instance:
uuid: "2f429bde36194fccbe8624caaa3b6b86"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.833868 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 02:06:30.837047 13272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.837759 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:30.838035 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
uuid: "2f429bde36194fccbe8624caaa3b6b86"
format_stamp: "Formatted at 2024-05-03 02:06:30 on dist-test-slave-fmt4"
I20240503 02:06:30.838290 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714701959428988-12500-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:30.854295 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:30.855508 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:30.857156 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:30.859380 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:30.859573 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.859812 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:30.859963 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:30.898664 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.3:46479
I20240503 02:06:30.898753 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.3:46479 every 8 connection(s)
I20240503 02:06:30.910838 13335 heartbeater.cc:347] Connected to a master server at 127.12.53.62:40893
I20240503 02:06:30.911171 13335 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:30.911870 13335 heartbeater.cc:511] Master 127.12.53.62:40893 requested a full tablet report, sending...
I20240503 02:06:30.913592 13059 ts_manager.cc:194] Registered new tserver with Master: 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:30.913699 12500 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012194887s
I20240503 02:06:30.914855 13059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58162
I20240503 02:06:30.916025 12500 test_util.cc:268] Using random seed: 722497443
I20240503 02:06:30.936625 13059 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58168:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:30.939389 13059 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:31.007732 13226 tablet_service.cc:1467] Processing CreateTablet for tablet 8395e11e913d4de29d3cd07692664a3c (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 02:06:31.009121 13226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8395e11e913d4de29d3cd07692664a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.020229 13293 tablet_service.cc:1467] Processing CreateTablet for tablet 7967a694f1614ee3bd3e0bc2e558a592 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 02:06:31.020063 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 887ec09cc01447a3912d5fa631258428 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 02:06:31.021579 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 377a299c253f4d6fab023232d6db92ef (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 02:06:31.021893 13223 tablet_service.cc:1467] Processing CreateTablet for tablet 096c192de7b540c08f05f6287097bda1 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 02:06:31.021600 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7967a694f1614ee3bd3e0bc2e558a592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.022927 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a299c253f4d6fab023232d6db92ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.023085 13223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c192de7b540c08f05f6287097bda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.024292 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 096c192de7b540c08f05f6287097bda1 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 02:06:31.025164 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887ec09cc01447a3912d5fa631258428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.025892 13222 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2d90103c624a74b9311910a8d06551 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 02:06:31.026520 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 377a299c253f4d6fab023232d6db92ef (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 02:06:31.027124 13222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2d90103c624a74b9311910a8d06551. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.027716 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a299c253f4d6fab023232d6db92ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.028733 13292 tablet_service.cc:1467] Processing CreateTablet for tablet ff10d87bcde049a1b122f1364316acdb (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 02:06:31.029990 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff10d87bcde049a1b122f1364316acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.029845 13221 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4d81de850f4f07a21ff56e03ec3923 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 02:06:31.031088 13221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4d81de850f4f07a21ff56e03ec3923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.034442 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 17b593d939484822907a1512922a5759 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 02:06:31.023046 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2d90103c624a74b9311910a8d06551 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 02:06:31.035796 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b593d939484822907a1512922a5759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.035669 13219 tablet_service.cc:1467] Processing CreateTablet for tablet 7967a694f1614ee3bd3e0bc2e558a592 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 02:06:31.036427 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c192de7b540c08f05f6287097bda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.033902 13220 tablet_service.cc:1467] Processing CreateTablet for tablet 887ec09cc01447a3912d5fa631258428 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 02:06:31.039409 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 096c192de7b540c08f05f6287097bda1 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 02:06:31.040253 13218 tablet_service.cc:1467] Processing CreateTablet for tablet ff10d87bcde049a1b122f1364316acdb (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 02:06:31.033926 13299 tablet_service.cc:1467] Processing CreateTablet for tablet 28648c3d8cc64d0689d0aa50d4a9709d (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 02:06:31.040688 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c192de7b540c08f05f6287097bda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.042435 13150 tablet_service.cc:1467] Processing CreateTablet for tablet 28648c3d8cc64d0689d0aa50d4a9709d (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 02:06:31.042677 13217 tablet_service.cc:1467] Processing CreateTablet for tablet 17b593d939484822907a1512922a5759 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 02:06:31.043701 13150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28648c3d8cc64d0689d0aa50d4a9709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.044528 13224 tablet_service.cc:1467] Processing CreateTablet for tablet 377a299c253f4d6fab023232d6db92ef (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 02:06:31.045310 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2d90103c624a74b9311910a8d06551. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.045951 13225 tablet_service.cc:1467] Processing CreateTablet for tablet 28648c3d8cc64d0689d0aa50d4a9709d (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 02:06:31.021922 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4d81de850f4f07a21ff56e03ec3923 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 02:06:31.035040 13300 tablet_service.cc:1467] Processing CreateTablet for tablet 8395e11e913d4de29d3cd07692664a3c (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 02:06:31.037909 13151 tablet_service.cc:1467] Processing CreateTablet for tablet 8395e11e913d4de29d3cd07692664a3c (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 02:06:31.049882 13151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8395e11e913d4de29d3cd07692664a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.054821 13219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7967a694f1614ee3bd3e0bc2e558a592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.055500 13217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b593d939484822907a1512922a5759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.062091 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4d81de850f4f07a21ff56e03ec3923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.064496 13147 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2d90103c624a74b9311910a8d06551 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 02:06:31.064963 13225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28648c3d8cc64d0689d0aa50d4a9709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.065625 13299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28648c3d8cc64d0689d0aa50d4a9709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.065840 13147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2d90103c624a74b9311910a8d06551. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.068555 13146 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4d81de850f4f07a21ff56e03ec3923 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 02:06:31.071141 13142 tablet_service.cc:1467] Processing CreateTablet for tablet 17b593d939484822907a1512922a5759 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 02:06:31.074879 13143 tablet_service.cc:1467] Processing CreateTablet for tablet ff10d87bcde049a1b122f1364316acdb (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 02:06:31.077841 13144 tablet_service.cc:1467] Processing CreateTablet for tablet 7967a694f1614ee3bd3e0bc2e558a592 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 02:06:31.080845 13145 tablet_service.cc:1467] Processing CreateTablet for tablet 887ec09cc01447a3912d5fa631258428 (DEFAULT_TABLE table=test-workload [id=26362bdea8ca4ebfb8f00a179a4a99de]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 02:06:31.082041 13224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377a299c253f4d6fab023232d6db92ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.082722 13218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff10d87bcde049a1b122f1364316acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.083117 13300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8395e11e913d4de29d3cd07692664a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.072604 13146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4d81de850f4f07a21ff56e03ec3923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.084761 13220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887ec09cc01447a3912d5fa631258428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.088620 13143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff10d87bcde049a1b122f1364316acdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.093456 13145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887ec09cc01447a3912d5fa631258428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.097077 13142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b593d939484822907a1512922a5759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.101059 13144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7967a694f1614ee3bd3e0bc2e558a592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:31.148501 13350 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.171154 13351 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.194787 13350 tablet_bootstrap.cc:654] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.195364 13351 tablet_bootstrap.cc:654] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.223348 13353 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.240665 13353 tablet_bootstrap.cc:654] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.251844 13350 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.252341 13350 ts_tablet_manager.cc:1367] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.104s user 0.017s sys 0.014s
I20240503 02:06:31.255017 13350 raft_consensus.cc:348] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.255743 13350 raft_consensus.cc:374] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.256148 13350 raft_consensus.cc:729] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.256933 13350 consensus_queue.cc:260] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.258241 13351 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.258710 13351 ts_tablet_manager.cc:1367] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.000s
I20240503 02:06:31.258991 13335 heartbeater.cc:503] Master 127.12.53.62:40893 was elected leader, sending a full tablet report...
I20240503 02:06:31.259471 13350 ts_tablet_manager.cc:1398] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 02:06:31.260362 13350 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.261286 13351 raft_consensus.cc:348] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.261937 13351 raft_consensus.cc:374] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.262123 13351 raft_consensus.cc:729] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.262737 13351 consensus_queue.cc:260] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.264281 13353 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.264775 13353 ts_tablet_manager.cc:1367] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.006s
I20240503 02:06:31.265919 13350 tablet_bootstrap.cc:654] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.267575 13353 raft_consensus.cc:348] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.268348 13353 raft_consensus.cc:374] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.268622 13353 raft_consensus.cc:729] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.269469 13353 consensus_queue.cc:260] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.271731 13351 ts_tablet_manager.cc:1398] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20240503 02:06:31.272531 13186 heartbeater.cc:503] Master 127.12.53.62:40893 was elected leader, sending a full tablet report...
I20240503 02:06:31.286685 13350 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.287171 13350 ts_tablet_manager.cc:1367] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.015s
I20240503 02:06:31.287981 13351 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.289753 13350 raft_consensus.cc:348] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.290467 13350 raft_consensus.cc:374] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.290771 13350 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.291510 13350 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.295430 13351 tablet_bootstrap.cc:654] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.296196 13350 ts_tablet_manager.cc:1398] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 02:06:31.297210 13350 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.299731 13261 heartbeater.cc:503] Master 127.12.53.62:40893 was elected leader, sending a full tablet report...
I20240503 02:06:31.300211 13353 ts_tablet_manager.cc:1398] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.035s user 0.018s sys 0.016s
I20240503 02:06:31.301118 13353 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.302815 13350 tablet_bootstrap.cc:654] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.306881 13353 tablet_bootstrap.cc:654] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.308439 13356 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.309002 13356 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.319725 13353 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.320256 13353 ts_tablet_manager.cc:1367] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20240503 02:06:31.322850 13353 raft_consensus.cc:348] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.323588 13353 raft_consensus.cc:374] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.323887 13353 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.324801 13353 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.326857 13353 ts_tablet_manager.cc:1398] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240503 02:06:31.330502 13351 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.330538 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.331307 13351 ts_tablet_manager.cc:1367] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.005s
W20240503 02:06:31.332553 13276 leader_election.cc:341] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.333336 13356 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.335294 13356 raft_consensus.cc:482] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.333973 13351 raft_consensus.cc:348] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.335852 13351 raft_consensus.cc:374] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.336589 13351 raft_consensus.cc:729] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.335896 13356 raft_consensus.cc:504] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.340075 13353 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.340130 13351 consensus_queue.cc:260] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.339921 13356 leader_election.cc:288] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.346161 13353 tablet_bootstrap.cc:654] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.348343 13351 ts_tablet_manager.cc:1398] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20240503 02:06:31.352005 13351 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.358389 13351 tablet_bootstrap.cc:654] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.359287 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.360049 13161 raft_consensus.cc:2454] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 0.
I20240503 02:06:31.362701 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.365092 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.366612 13276 leader_election.cc:302] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:31.367547 13356 raft_consensus.cc:2781] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.367981 13356 raft_consensus.cc:482] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.368409 13356 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.370443 13351 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.371073 13351 ts_tablet_manager.cc:1367] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.008s
W20240503 02:06:31.371732 13276 leader_election.cc:341] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.374022 13351 raft_consensus.cc:348] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.377202 13351 raft_consensus.cc:374] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.376965 13276 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86; no voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:31.377586 13351 raft_consensus.cc:729] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.378239 13363 raft_consensus.cc:2726] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:31.378520 13351 consensus_queue.cc:260] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.379074 13356 raft_consensus.cc:504] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.381647 13356 leader_election.cc:288] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
W20240503 02:06:31.378609 13276 leader_election.cc:341] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.383672 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.384310 13161 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.389232 13351 ts_tablet_manager.cc:1398] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20240503 02:06:31.390084 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.390398 13351 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
W20240503 02:06:31.393182 13276 leader_election.cc:341] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.397169 13350 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.397657 13350 ts_tablet_manager.cc:1367] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.101s user 0.035s sys 0.049s
I20240503 02:06:31.397800 13161 raft_consensus.cc:2454] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 1.
I20240503 02:06:31.398231 13353 tablet_bootstrap.cc:492] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.398722 13353 ts_tablet_manager.cc:1367] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.059s user 0.023s sys 0.031s
I20240503 02:06:31.399150 13276 leader_election.cc:302] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters: ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:31.399987 13363 raft_consensus.cc:2781] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.400415 13351 tablet_bootstrap.cc:654] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.401247 13353 raft_consensus.cc:348] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.402000 13353 raft_consensus.cc:374] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.402302 13353 raft_consensus.cc:729] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.402980 13353 consensus_queue.cc:260] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.405362 13350 raft_consensus.cc:348] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.406219 13350 raft_consensus.cc:374] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.406567 13350 raft_consensus.cc:729] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.407424 13350 consensus_queue.cc:260] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.408390 13353 ts_tablet_manager.cc:1398] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240503 02:06:31.409376 13353 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.410079 13350 ts_tablet_manager.cc:1398] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240503 02:06:31.411314 13350 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.413209 13363 raft_consensus.cc:686] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:31.414047 13363 consensus_queue.cc:237] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.414932 13353 tablet_bootstrap.cc:654] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.417970 13351 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.418087 13350 tablet_bootstrap.cc:654] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.418514 13351 ts_tablet_manager.cc:1367] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.001s
I20240503 02:06:31.422677 13351 raft_consensus.cc:348] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.423384 13351 raft_consensus.cc:374] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.423465 13353 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.423741 13351 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.424093 13353 ts_tablet_manager.cc:1367] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20240503 02:06:31.424768 13351 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.427244 13351 ts_tablet_manager.cc:1398] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20240503 02:06:31.428145 13351 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.435948 13058 catalog_manager.cc:5484] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:31.437384 13351 tablet_bootstrap.cc:654] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.429235 13353 raft_consensus.cc:348] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.438627 13353 raft_consensus.cc:374] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.438949 13353 raft_consensus.cc:729] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.439801 13353 consensus_queue.cc:260] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.442998 13353 ts_tablet_manager.cc:1398] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.019s user 0.001s sys 0.005s
I20240503 02:06:31.443943 13353 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.447240 13350 tablet_bootstrap.cc:492] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.447767 13350 ts_tablet_manager.cc:1367] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.007s
I20240503 02:06:31.450763 13353 tablet_bootstrap.cc:654] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.450485 13350 raft_consensus.cc:348] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.451529 13350 raft_consensus.cc:374] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.451932 13350 raft_consensus.cc:729] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.452685 13350 consensus_queue.cc:260] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.455121 13350 ts_tablet_manager.cc:1398] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 02:06:31.456027 13350 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.458964 13351 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.459524 13351 ts_tablet_manager.cc:1367] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20240503 02:06:31.462368 13350 tablet_bootstrap.cc:654] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.462760 13351 raft_consensus.cc:348] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.463470 13351 raft_consensus.cc:374] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.463748 13353 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.463773 13351 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.464326 13353 ts_tablet_manager.cc:1367] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.004s
I20240503 02:06:31.464674 13351 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.466490 13350 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.466840 13350 ts_tablet_manager.cc:1367] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 02:06:31.466951 13353 raft_consensus.cc:348] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.467651 13353 raft_consensus.cc:374] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.467926 13353 raft_consensus.cc:729] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.469146 13351 ts_tablet_manager.cc:1398] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:06:31.468721 13353 consensus_queue.cc:260] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.469043 13350 raft_consensus.cc:348] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.469807 13350 raft_consensus.cc:374] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.470120 13350 raft_consensus.cc:729] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.470156 13351 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.470886 13350 consensus_queue.cc:260] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.473106 13350 ts_tablet_manager.cc:1398] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:06:31.473963 13350 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.475075 13353 ts_tablet_manager.cc:1398] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240503 02:06:31.475967 13353 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.477941 13351 tablet_bootstrap.cc:654] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.479343 13350 tablet_bootstrap.cc:654] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.481458 13353 tablet_bootstrap.cc:654] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.486295 13359 raft_consensus.cc:482] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.486788 13359 raft_consensus.cc:504] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.489244 13359 leader_election.cc:288] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.489260 13351 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.489903 13351 ts_tablet_manager.cc:1367] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20240503 02:06:31.490674 13350 tablet_bootstrap.cc:492] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.491145 13350 ts_tablet_manager.cc:1367] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240503 02:06:31.492578 13351 raft_consensus.cc:348] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.493332 13351 raft_consensus.cc:374] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.493643 13351 raft_consensus.cc:729] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.493755 13350 raft_consensus.cc:348] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.494630 13350 raft_consensus.cc:374] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.494411 13351 consensus_queue.cc:260] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.494939 13350 raft_consensus.cc:729] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.495748 13350 consensus_queue.cc:260] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.498040 13350 ts_tablet_manager.cc:1398] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:06:31.498550 13351 ts_tablet_manager.cc:1398] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:06:31.498966 13350 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.499555 13351 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.505008 13350 tablet_bootstrap.cc:654] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.516119 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.516868 13161 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.518231 13202 leader_election.cc:302] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.519029 13359 raft_consensus.cc:2781] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.519261 13350 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.519393 13359 raft_consensus.cc:482] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.519727 13350 ts_tablet_manager.cc:1367] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20240503 02:06:31.519752 13359 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.520772 13351 tablet_bootstrap.cc:654] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.522405 13350 raft_consensus.cc:348] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.523258 13350 raft_consensus.cc:374] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.523977 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.524637 13310 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.526648 13353 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.527168 13353 ts_tablet_manager.cc:1367] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.005s
I20240503 02:06:31.528064 13359 raft_consensus.cc:504] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.523528 13350 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.529983 13353 raft_consensus.cc:348] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.530282 13350 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.531038 13353 raft_consensus.cc:374] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.530854 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.531450 13353 raft_consensus.cc:729] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.531638 13161 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.532300 13353 consensus_queue.cc:260] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.536115 13350 ts_tablet_manager.cc:1398] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20240503 02:06:31.537114 13350 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.539427 13161 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.540360 13353 ts_tablet_manager.cc:1398] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20240503 02:06:31.540761 13202 leader_election.cc:302] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.541276 13353 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.545051 13351 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.545524 13351 ts_tablet_manager.cc:1367] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.000s
I20240503 02:06:31.546204 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.546687 13350 tablet_bootstrap.cc:654] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.546782 13310 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.545225 13359 leader_election.cc:288] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.549213 13359 raft_consensus.cc:2781] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.549650 13359 raft_consensus.cc:686] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:31.551023 13351 raft_consensus.cc:348] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.551817 13351 raft_consensus.cc:374] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.552090 13351 raft_consensus.cc:729] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.552824 13359 consensus_queue.cc:237] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.554946 13351 consensus_queue.cc:260] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.556906 13310 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.538605 13383 raft_consensus.cc:482] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.559165 13351 ts_tablet_manager.cc:1398] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20240503 02:06:31.558727 13383 raft_consensus.cc:504] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.560180 13351 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.561076 13350 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.561527 13350 ts_tablet_manager.cc:1367] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20240503 02:06:31.561666 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
W20240503 02:06:31.561599 13107 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:06:31.562489 13161 raft_consensus.cc:2379] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef52048cb739452e9b4b2266943ec12f in current term 1: Already voted for candidate 2f429bde36194fccbe8624caaa3b6b86 in this term.
I20240503 02:06:31.566123 13351 tablet_bootstrap.cc:654] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.566192 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.566489 13350 raft_consensus.cc:348] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.567154 13350 raft_consensus.cc:374] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.567405 13350 raft_consensus.cc:729] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.567730 13200 leader_election.cc:302] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef52048cb739452e9b4b2266943ec12f; no voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:31.568094 13350 consensus_queue.cc:260] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.570796 13383 leader_election.cc:288] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.571545 13383 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.572127 13350 ts_tablet_manager.cc:1398] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20240503 02:06:31.572995 13350 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.577776 13350 tablet_bootstrap.cc:654] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.578608 13383 raft_consensus.cc:2726] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:31.579293 13383 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.579707 13383 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.581051 13353 tablet_bootstrap.cc:654] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.585903 13351 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.586382 13351 ts_tablet_manager.cc:1367] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.005s
I20240503 02:06:31.588151 13350 tablet_bootstrap.cc:492] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.588515 13350 ts_tablet_manager.cc:1367] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240503 02:06:31.588973 13363 raft_consensus.cc:482] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.589435 13363 raft_consensus.cc:504] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.590126 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.590384 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.590870 13161 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.591266 13363 leader_election.cc:288] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.592140 13383 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.592104 13202 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.592895 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.593437 13383 raft_consensus.cc:2781] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.593842 13383 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.593863 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.594295 13056 catalog_manager.cc:5484] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 1 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.593514 13161 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 0.
W20240503 02:06:31.595876 13276 leader_election.cc:341] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.596486 13383 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.596880 13276 leader_election.cc:302] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters: ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:31.597813 13363 raft_consensus.cc:2781] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.598171 13363 raft_consensus.cc:482] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.598488 13363 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.603227 13383 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.606170 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.606799 13161 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.604645 13363 raft_consensus.cc:504] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
W20240503 02:06:31.607728 13200 leader_election.cc:341] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.608155 13351 raft_consensus.cc:348] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.608811 13351 raft_consensus.cc:374] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.609071 13351 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.609659 13351 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.610103 13353 tablet_bootstrap.cc:492] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.609977 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.610586 13353 ts_tablet_manager.cc:1367] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.012s
I20240503 02:06:31.610620 13160 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.613602 13161 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.613755 13351 ts_tablet_manager.cc:1398] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20240503 02:06:31.613389 13353 raft_consensus.cc:348] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.614266 13353 raft_consensus.cc:374] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.614552 13353 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.614800 13351 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.614946 13202 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.615290 13353 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.615828 13359 raft_consensus.cc:2781] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.616340 13359 raft_consensus.cc:686] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:31.616443 13383 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.617228 13359 consensus_queue.cc:237] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.618322 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.620419 13353 ts_tablet_manager.cc:1398] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20240503 02:06:31.620443 13160 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 1.
I20240503 02:06:31.621544 13353 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
W20240503 02:06:31.621886 13200 leader_election.cc:341] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.622211 13363 leader_election.cc:288] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.624433 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.621616 13350 raft_consensus.cc:348] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.626410 13350 raft_consensus.cc:374] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.622928 13276 leader_election.cc:302] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:31.626683 13350 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
W20240503 02:06:31.627207 13276 leader_election.cc:341] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.627596 13353 tablet_bootstrap.cc:654] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.627285 13350 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.628410 13367 raft_consensus.cc:2781] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.628415 13357 raft_consensus.cc:482] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.629165 13357 raft_consensus.cc:504] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.631111 13357 leader_election.cc:288] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.632423 13351 tablet_bootstrap.cc:654] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.633582 13353 tablet_bootstrap.cc:492] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.634330 13353 ts_tablet_manager.cc:1367] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20240503 02:06:31.637281 13353 raft_consensus.cc:348] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.638044 13353 raft_consensus.cc:374] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.638360 13353 raft_consensus.cc:729] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.638816 13367 raft_consensus.cc:686] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:31.639202 13353 consensus_queue.cc:260] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.639689 13367 consensus_queue.cc:237] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.641938 13353 ts_tablet_manager.cc:1398] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:06:31.642864 13353 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.646703 13357 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.647154 13357 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.649086 13357 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.649220 13353 tablet_bootstrap.cc:654] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.649577 13357 raft_consensus.cc:482] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.650058 13357 raft_consensus.cc:504] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.652454 13357 leader_election.cc:288] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.657049 13359 raft_consensus.cc:482] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.657630 13359 raft_consensus.cc:504] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.651732 13056 catalog_manager.cc:5484] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 1 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.660611 13359 leader_election.cc:288] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.660857 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.657641 13350 ts_tablet_manager.cc:1398] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.069s user 0.006s sys 0.000s
I20240503 02:06:31.661512 13310 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.661976 13351 tablet_bootstrap.cc:492] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.663427 13351 ts_tablet_manager.cc:1367] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20240503 02:06:31.664366 13353 tablet_bootstrap.cc:492] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.662861 13350 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: Bootstrap starting.
I20240503 02:06:31.662654 13200 leader_election.cc:302] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.664852 13353 ts_tablet_manager.cc:1367] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20240503 02:06:31.667235 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.667632 13353 raft_consensus.cc:348] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.668323 13353 raft_consensus.cc:374] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.668244 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.668910 13359 raft_consensus.cc:2781] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.669057 13235 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.669297 13359 raft_consensus.cc:482] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.669673 13359 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.668699 13353 raft_consensus.cc:729] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.671140 13353 consensus_queue.cc:260] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.673379 13353 ts_tablet_manager.cc:1398] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:06:31.674237 13353 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: Bootstrap starting.
I20240503 02:06:31.673938 13351 raft_consensus.cc:348] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.676548 13351 raft_consensus.cc:374] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.676918 13351 raft_consensus.cc:729] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.677084 13059 catalog_manager.cc:5484] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:31.679662 13353 tablet_bootstrap.cc:654] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.679672 13350 tablet_bootstrap.cc:654] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.677593 13351 consensus_queue.cc:260] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.687139 13351 ts_tablet_manager.cc:1398] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240503 02:06:31.688179 13351 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: Bootstrap starting.
I20240503 02:06:31.691833 13127 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.693526 13396 raft_consensus.cc:2781] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.685087 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8395e11e913d4de29d3cd07692664a3c" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.694026 13396 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.694394 13396 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.694350 13234 raft_consensus.cc:2454] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.692716 13236 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.701828 13359 raft_consensus.cc:504] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.705652 13127 leader_election.cc:302] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.707226 13357 raft_consensus.cc:2781] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.707633 13357 raft_consensus.cc:482] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.708806 13127 leader_election.cc:302] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.708801 13359 leader_election.cc:288] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.708220 13357 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.715878 13396 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.710707 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.716986 13310 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.717140 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.717880 13160 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.718380 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.718963 13161 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.720160 13357 raft_consensus.cc:504] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.721268 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.721930 13309 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.722198 13357 leader_election.cc:288] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.723765 13357 raft_consensus.cc:2781] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.724287 13357 raft_consensus.cc:482] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.724658 13357 raft_consensus.cc:3037] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.727013 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.727560 13309 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.729730 13396 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.724440 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.730631 13308 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.731323 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.723728 13307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8395e11e913d4de29d3cd07692664a3c" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.731918 13306 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.732965 13353 tablet_bootstrap.cc:492] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: No bootstrap required, opened a new log
I20240503 02:06:31.733381 13353 ts_tablet_manager.cc:1367] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.004s
I20240503 02:06:31.735230 13350 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: No bootstrap required, opened a new log
I20240503 02:06:31.735704 13350 ts_tablet_manager.cc:1367] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.020s
I20240503 02:06:31.735895 13307 raft_consensus.cc:2454] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 0.
I20240503 02:06:31.735706 13353 raft_consensus.cc:348] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.736956 13353 raft_consensus.cc:374] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.737232 13353 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.740424 13350 raft_consensus.cc:348] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.741454 13350 raft_consensus.cc:374] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.741784 13350 raft_consensus.cc:729] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.741852 13309 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.738945 13357 raft_consensus.cc:504] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.742434 13161 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.742856 13353 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.744292 13350 consensus_queue.cc:260] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.745051 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.745633 13202 leader_election.cc:302] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.746079 13234 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.746556 13350 ts_tablet_manager.cc:1398] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:06:31.748920 13359 raft_consensus.cc:2781] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.749418 13359 raft_consensus.cc:686] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:31.750540 13367 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.750721 13359 consensus_queue.cc:237] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.751003 13367 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.752827 13367 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.743708 13357 leader_election.cc:288] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.758378 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.759019 13236 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 0.
I20240503 02:06:31.770012 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.770684 13236 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.771644 13234 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.773160 13276 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.774138 13367 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.774524 13367 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.774853 13367 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.784540 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8395e11e913d4de29d3cd07692664a3c" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.785313 13309 raft_consensus.cc:3037] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.787191 13367 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.806582 13236 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.808360 13367 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.804541 13309 raft_consensus.cc:2454] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.809666 13306 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.813786 13310 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.843281 13059 catalog_manager.cc:5484] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 1 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.849417 13353 ts_tablet_manager.cc:1398] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: Time spent starting tablet: real 0.116s user 0.007s sys 0.000s
I20240503 02:06:31.851087 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.853992 13351 tablet_bootstrap.cc:654] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:31.854259 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.861274 13367 consensus_queue.cc:1035] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 02:06:31.869561 13236 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.869645 13359 raft_consensus.cc:482] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.870247 13359 raft_consensus.cc:504] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.853895 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8395e11e913d4de29d3cd07692664a3c" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:31.872159 13359 leader_election.cc:288] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.857766 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.873261 13161 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 0.
I20240503 02:06:31.851884 13127 leader_election.cc:302] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
W20240503 02:06:31.874135 13125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.3:46479, user_credentials={real_user=slave}} blocked reactor thread for 46429.2us
I20240503 02:06:31.871953 13234 raft_consensus.cc:3037] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.874504 13357 raft_consensus.cc:2781] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.874650 13127 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.875195 13357 raft_consensus.cc:686] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:31.876083 13357 consensus_queue.cc:237] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.876631 13396 raft_consensus.cc:2781] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.877126 13396 raft_consensus.cc:686] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:31.877889 13396 consensus_queue.cc:237] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.873829 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86" is_pre_election: true
I20240503 02:06:31.880046 13310 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 0.
I20240503 02:06:31.881073 13200 leader_election.cc:302] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.882001 13359 raft_consensus.cc:2781] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:31.882328 13359 raft_consensus.cc:482] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:31.882679 13359 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.884342 13367 consensus_queue.cc:1035] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:31.886261 13125 leader_election.cc:302] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:31.887214 13400 raft_consensus.cc:2781] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.888603 13400 raft_consensus.cc:686] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:31.889545 13400 consensus_queue.cc:237] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.893410 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.894343 13359 raft_consensus.cc:504] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
W20240503 02:06:31.895090 13202 leader_election.cc:341] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:31.898321 13236 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 1.
I20240503 02:06:31.900067 13351 tablet_bootstrap.cc:492] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: No bootstrap required, opened a new log
I20240503 02:06:31.898697 13234 raft_consensus.cc:2454] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 1.
I20240503 02:06:31.900537 13351 ts_tablet_manager.cc:1367] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: Time spent bootstrapping tablet: real 0.213s user 0.049s sys 0.081s
I20240503 02:06:31.901621 13276 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters: d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:31.902647 13363 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:31.903153 13363 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.903860 13351 raft_consensus.cc:348] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.904558 13351 raft_consensus.cc:374] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:31.904832 13351 raft_consensus.cc:729] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Initialized, Role: FOLLOWER
I20240503 02:06:31.905041 13363 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:31.905556 13351 consensus_queue.cc:260] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.905771 13059 catalog_manager.cc:5484] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.907902 13351 ts_tablet_manager.cc:1398] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:06:31.909644 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878" is_pre_election: true
I20240503 02:06:31.905570 13363 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.910341 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef52048cb739452e9b4b2266943ec12f" is_pre_election: true
I20240503 02:06:31.910387 13160 raft_consensus.cc:2379] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f429bde36194fccbe8624caaa3b6b86 in current term 1: Already voted for candidate ef52048cb739452e9b4b2266943ec12f in this term.
I20240503 02:06:31.911115 13363 raft_consensus.cc:686] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:31.911026 13159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:31.912286 13276 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86; no voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:31.912042 13363 consensus_queue.cc:237] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.912598 13159 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.913480 13356 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.914805 13359 leader_election.cc:288] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:31.919437 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:31.920001 13310 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:31.935209 13159 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.934567 13055 catalog_manager.cc:5484] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:31.935727 13356 raft_consensus.cc:2726] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:31.944000 13202 leader_election.cc:302] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:31.945335 13359 raft_consensus.cc:2781] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:31.945802 13359 raft_consensus.cc:686] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:31.946570 13359 consensus_queue.cc:237] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:31.952095 13384 consensus_queue.cc:1035] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:31.943833 13310 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 1.
I20240503 02:06:31.965584 13056 catalog_manager.cc:5484] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.967283 13056 catalog_manager.cc:5484] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.980716 13059 catalog_manager.cc:5484] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 1 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:31.985956 13383 consensus_queue.cc:1035] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.037463 13235 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:32.043054 13356 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.052783 13367 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.077998 13383 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.087976 13383 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.180364 13235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:32.181049 13235 raft_consensus.cc:595] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:32.187459 13235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:32.187916 13235 raft_consensus.cc:595] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:32.198956 13306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:57264
I20240503 02:06:32.199460 13306 raft_consensus.cc:595] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:32.211501 13159 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:60764
I20240503 02:06:32.212002 13159 raft_consensus.cc:595] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Received request to transfer leadership to TS ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:32.216398 13306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "887ec09cc01447a3912d5fa631258428"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
mode: GRACEFUL
new_leader_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:57264
I20240503 02:06:32.216913 13306 raft_consensus.cc:595] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Received request to transfer leadership to TS ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:32.223222 13306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
mode: GRACEFUL
new_leader_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:57264
I20240503 02:06:32.223762 13306 raft_consensus.cc:595] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Received request to transfer leadership to TS ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:32.227787 13235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17b593d939484822907a1512922a5759"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:32.228267 13235 raft_consensus.cc:595] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Received request to transfer leadership to TS 2f429bde36194fccbe8624caaa3b6b86
I20240503 02:06:32.293489 13383 consensus_queue.cc:1035] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.303258 13384 consensus_queue.cc:1035] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.309777 13383 raft_consensus.cc:979] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: : Instructing follower 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:32.310187 13383 raft_consensus.cc:1067] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Signalling peer 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:32.311457 13306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17b593d939484822907a1512922a5759"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:57236
I20240503 02:06:32.311923 13306 raft_consensus.cc:482] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.312192 13306 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.316545 13306 raft_consensus.cc:504] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.318128 13306 leader_election.cc:288] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 2 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:32.319116 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:32.319443 13159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b593d939484822907a1512922a5759" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:32.319717 13235 raft_consensus.cc:3032] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.320019 13235 raft_consensus.cc:729] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.320077 13159 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.320628 13235 consensus_queue.cc:260] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.321592 13235 raft_consensus.cc:3037] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.326200 13159 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 2.
I20240503 02:06:32.326833 13235 raft_consensus.cc:2454] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 2.
I20240503 02:06:32.327097 13276 leader_election.cc:302] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:32.327724 13367 raft_consensus.cc:2781] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.328094 13367 raft_consensus.cc:686] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:32.328778 13367 consensus_queue.cc:237] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.334371 13055 catalog_manager.cc:5484] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 1 to 2, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 2 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:32.339530 13357 consensus_queue.cc:1035] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.349946 13400 consensus_queue.cc:1035] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.361624 13384 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.369130 13367 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.374390 13396 consensus_queue.cc:1035] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.375067 13384 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.388578 13159 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:32.393363 13400 consensus_queue.cc:1035] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.394659 13357 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.400204 13367 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.415457 13359 raft_consensus.cc:979] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.415822 13396 raft_consensus.cc:979] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.415910 13359 raft_consensus.cc:1067] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.416359 13396 raft_consensus.cc:1067] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Signalling peer ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.416906 13363 raft_consensus.cc:979] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.417338 13363 raft_consensus.cc:1067] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.419318 13161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60748
I20240503 02:06:32.419778 13161 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.420085 13161 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.420431 13357 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:32.425715 13160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60752
I20240503 02:06:32.425927 13236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:45334
I20240503 02:06:32.426290 13160 raft_consensus.cc:482] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.426491 13236 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.426685 13160 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.426950 13236 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.426652 13161 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.434231 13160 raft_consensus.cc:504] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.434862 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.435633 13306 raft_consensus.cc:3032] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.436034 13306 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:32.437049 13161 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.436946 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:32.437747 13233 raft_consensus.cc:3032] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.438113 13233 raft_consensus.cc:729] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.437918 13236 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.438905 13233 consensus_queue.cc:260] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.439476 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:32.439755 13235 raft_consensus.cc:1226] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Rejecting Update request from peer d6ed2caa226f4e12ac39799a6e80e878 for earlier term 1. Current term is 2. Ops: []
I20240503 02:06:32.436981 13306 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.440639 13356 raft_consensus.cc:979] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: : Instructing follower ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.442591 13356 raft_consensus.cc:1067] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Signalling peer ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.443012 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28648c3d8cc64d0689d0aa50d4a9709d" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.443573 13310 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.444031 13236 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.446597 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.447166 13309 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.448011 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:32.448236 13236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:45332
I20240503 02:06:32.449028 13236 raft_consensus.cc:482] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.449479 13236 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.448585 13161 raft_consensus.cc:3032] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.450244 13161 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:32.452244 13161 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.447943 13160 leader_election.cc:288] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.453581 13161 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.455673 13306 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.456768 13236 raft_consensus.cc:504] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.440018 13234 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.440049 13233 raft_consensus.cc:3037] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.459245 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:32.459762 13160 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.459246 13307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff10d87bcde049a1b122f1364316acdb" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.461756 13307 raft_consensus.cc:3032] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.461997 13307 raft_consensus.cc:729] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:32.462592 13307 consensus_queue.cc:260] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.463487 13307 raft_consensus.cc:3037] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.459641 13309 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.465711 13233 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.466671 13200 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:32.466622 13127 leader_election.cc:302] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:32.467609 13382 raft_consensus.cc:2781] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.468076 13382 raft_consensus.cc:686] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.468884 13310 raft_consensus.cc:2454] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.468878 13382 consensus_queue.cc:237] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.469717 13307 raft_consensus.cc:2454] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.470257 13160 raft_consensus.cc:2454] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.470860 13200 leader_election.cc:302] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:32.471753 13384 raft_consensus.cc:2781] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.467609 13357 raft_consensus.cc:2781] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.472333 13384 raft_consensus.cc:686] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.471834 13161 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.473732 13236 leader_election.cc:288] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.473235 13384 consensus_queue.cc:237] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.473032 13357 raft_consensus.cc:686] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:32.475184 13357 consensus_queue.cc:237] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.478967 13234 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.480417 13367 raft_consensus.cc:979] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: : Instructing follower ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.480268 13127 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:32.481029 13367 raft_consensus.cc:1067] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Signalling peer ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:32.481273 13400 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.481770 13400 raft_consensus.cc:686] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:32.482779 13306 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.482556 13400 consensus_queue.cc:237] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.482960 13055 catalog_manager.cc:5484] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 1 to 2, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.484733 13234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "887ec09cc01447a3912d5fa631258428"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:45332
I20240503 02:06:32.485466 13234 raft_consensus.cc:482] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.486673 13234 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.488794 13059 catalog_manager.cc:5484] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 1 to 2, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 2 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.490417 13059 catalog_manager.cc:5484] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 1 to 2, leader changed from 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3) to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 2 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.493242 13234 raft_consensus.cc:504] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.495671 13234 leader_election.cc:288] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.496304 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.496960 13306 raft_consensus.cc:3032] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.497187 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:32.497298 13306 raft_consensus.cc:729] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:32.497790 13161 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.497380 13055 catalog_manager.cc:5484] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 1 to 2, leader changed from 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.498173 13306 consensus_queue.cc:260] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.499652 13306 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.504393 13161 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.505564 13202 leader_election.cc:302] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:32.506309 13306 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 2.
I20240503 02:06:32.506356 13384 raft_consensus.cc:2781] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.506975 13384 raft_consensus.cc:686] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.507643 13384 consensus_queue.cc:237] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.513141 13055 catalog_manager.cc:5484] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 1 to 2, leader changed from 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3) to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 2 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.699877 13382 raft_consensus.cc:979] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.700301 13384 raft_consensus.cc:1067] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:32.701624 13161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60752
I20240503 02:06:32.702059 13161 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:32.702348 13161 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.706956 13161 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.708364 13161 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:32.709147 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:32.709329 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:32.709760 13234 raft_consensus.cc:3032] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:32.709825 13306 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.710220 13234 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:32.711015 13234 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.712318 13234 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:32.716466 13306 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.717643 13125 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:32.718326 13234 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:32.718503 13400 raft_consensus.cc:2781] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:32.718971 13400 raft_consensus.cc:686] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:32.719759 13400 consensus_queue.cc:237] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:32.725756 13055 catalog_manager.cc:5484] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 1 to 2, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:32.837514 13161 raft_consensus.cc:1261] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.838711 13356 consensus_queue.cc:1035] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.845446 13234 raft_consensus.cc:1261] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.847247 13363 consensus_queue.cc:1035] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.891997 13306 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.893172 13384 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.900533 13161 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.902174 13384 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.955006 13161 raft_consensus.cc:1261] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.956238 13384 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.958527 13234 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.959861 13357 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.975399 13306 raft_consensus.cc:1261] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.977030 13359 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:32.983213 13306 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:32.984581 13396 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.002187 13306 raft_consensus.cc:1261] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.003273 13384 consensus_queue.cc:1035] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.011005 13161 raft_consensus.cc:1261] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.012765 13437 consensus_queue.cc:1035] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.041216 13234 raft_consensus.cc:1261] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.042538 13396 consensus_queue.cc:1035] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.053690 13306 raft_consensus.cc:1261] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.055083 13400 consensus_queue.cc:1035] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.124469 13234 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.125628 13400 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:33.133299 13306 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:33.134647 13396 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:35.251593 12500 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240503 02:06:35.252336 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:35.275158 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:35.275705 13161 raft_consensus.cc:595] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Received request to transfer leadership to TS 2f429bde36194fccbe8624caaa3b6b86
I20240503 02:06:35.278141 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:35.278617 13161 raft_consensus.cc:595] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Received request to transfer leadership to TS 2f429bde36194fccbe8624caaa3b6b86
I20240503 02:06:35.279503 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:06:35.279898 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:35.535228 13447 raft_consensus.cc:979] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:35.535660 13447 raft_consensus.cc:1067] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Signalling peer 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:35.537027 13306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:57252
I20240503 02:06:35.537509 13306 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:35.537796 13306 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.542332 13306 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.543727 13306 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 3 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:35.544602 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:35.544704 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:35.545192 13161 raft_consensus.cc:3032] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:35.545226 13234 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.545507 13161 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:35.546160 13161 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.547097 13161 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.549804 13234 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 3.
I20240503 02:06:35.550865 13276 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:35.551637 13450 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:35.552212 13161 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 3.
I20240503 02:06:35.553158 13450 raft_consensus.cc:686] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:35.553980 13450 consensus_queue.cc:237] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.560475 13055 catalog_manager.cc:5484] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 2 to 3, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 3 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:35.590323 13396 raft_consensus.cc:979] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:35.590761 13447 raft_consensus.cc:1067] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Signalling peer 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:35.592083 13306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:57252
I20240503 02:06:35.592528 13306 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:35.592834 13306 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.597148 13306 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.598549 13306 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 3 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:35.599457 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:35.599794 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:35.600091 13161 raft_consensus.cc:3032] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:35.600441 13234 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.600469 13161 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:35.601459 13161 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.602826 13161 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:35.607223 13234 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 3.
I20240503 02:06:35.608369 13276 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:35.609230 13450 raft_consensus.cc:2781] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:35.609334 13161 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 3.
I20240503 02:06:35.609776 13450 raft_consensus.cc:686] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:35.610667 13450 consensus_queue.cc:237] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:35.617429 13055 catalog_manager.cc:5484] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 2 to 3, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 3 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:35.982928 13161 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:35.984045 13450 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:35.993791 13234 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:35.995364 13450 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:35.996446 13236 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:35.999919 13450 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:36.012058 13160 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:36.013760 13450 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:36.280649 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:36.285881 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:36.286288 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:37.287039 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:37.292318 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:37.292781 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:38.293756 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:38.299525 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:38.299947 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:39.300808 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:39.306058 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:39.306442 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:40.307454 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:40.313354 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:40.313735 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:41.314603 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:41.320307 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:41.320801 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:42.321653 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:42.326735 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:42.327267 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:43.328054 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:43.333696 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:43.334131 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:44.334980 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:44.341286 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:44.341694 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:45.368196 12500 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240503 02:06:45.378718 13309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:57264
I20240503 02:06:45.379489 13309 raft_consensus.cc:595] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:45.384483 13309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:57264
I20240503 02:06:45.384961 13309 raft_consensus.cc:595] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:45.389153 13234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:45.389602 13234 raft_consensus.cc:595] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:45.393895 13234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "887ec09cc01447a3912d5fa631258428"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:45.394289 13234 raft_consensus.cc:595] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:45.398135 13234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
mode: GRACEFUL
new_leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:45342
I20240503 02:06:45.398524 13234 raft_consensus.cc:595] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Received request to transfer leadership to TS d6ed2caa226f4e12ac39799a6e80e878
I20240503 02:06:45.481065 13471 raft_consensus.cc:979] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.481607 13478 raft_consensus.cc:1067] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.483031 13160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60752
I20240503 02:06:45.483469 13160 raft_consensus.cc:482] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:45.483781 13160 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:45.488924 13160 raft_consensus.cc:504] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.490504 13160 leader_election.cc:288] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:45.491345 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:45.491608 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:45.492138 13234 raft_consensus.cc:3032] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:45.492218 13309 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:45.492391 13234 raft_consensus.cc:729] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:45.493741 13471 raft_consensus.cc:979] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.493238 13234 consensus_queue.cc:260] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.494208 13478 raft_consensus.cc:1067] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.494700 13234 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:45.495683 13160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "887ec09cc01447a3912d5fa631258428"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60752
I20240503 02:06:45.496191 13160 raft_consensus.cc:482] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:45.496665 13160 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.498785 13309 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:45.499663 13234 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 2.
I20240503 02:06:45.500161 13125 leader_election.cc:302] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:45.502015 13476 raft_consensus.cc:2781] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:45.503142 13476 raft_consensus.cc:686] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:45.503660 13475 raft_consensus.cc:979] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.502709 13160 raft_consensus.cc:504] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.504225 13477 raft_consensus.cc:1067] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.504073 13476 consensus_queue.cc:237] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.505669 13160 leader_election.cc:288] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 3 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:45.506353 13161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60748
I20240503 02:06:45.506314 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:45.506692 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887ec09cc01447a3912d5fa631258428" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:45.507182 13234 raft_consensus.cc:3032] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:45.507102 13309 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.507061 13161 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:45.508020 13161 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.507570 13234 raft_consensus.cc:729] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:45.510759 13234 consensus_queue.cc:260] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.511951 13234 raft_consensus.cc:3037] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.513453 13055 catalog_manager.cc:5484] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 1 to 2, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:45.514453 13309 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 3.
I20240503 02:06:45.515345 13161 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.517743 13161 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 4 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:45.517755 13125 leader_election.cc:302] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:45.518085 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:45.518457 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:45.518720 13236 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.519076 13309 raft_consensus.cc:3032] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:45.517431 13234 raft_consensus.cc:2454] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 3.
I20240503 02:06:45.519065 13476 raft_consensus.cc:2781] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:45.519613 13309 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:45.520212 13476 raft_consensus.cc:686] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:45.520634 13309 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.521229 13476 consensus_queue.cc:237] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.522262 13309 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.525727 13236 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 4.
I20240503 02:06:45.527329 13127 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:45.528965 13447 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:45.530843 13309 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 4.
I20240503 02:06:45.537030 13447 raft_consensus.cc:686] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:45.538213 13447 consensus_queue.cc:237] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.542449 13475 raft_consensus.cc:979] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.542841 13477 raft_consensus.cc:1067] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.544340 13161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "096c192de7b540c08f05f6287097bda1"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60748
I20240503 02:06:45.544883 13161 raft_consensus.cc:482] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:45.545121 13161 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.548337 13055 catalog_manager.cc:5484] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 2 to 3, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 3 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:45.549079 13161 raft_consensus.cc:504] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.549464 13055 catalog_manager.cc:5484] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 3 to 4, leader changed from 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 4 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:45.550619 13161 leader_election.cc:288] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 4 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:45.551285 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:45.551832 13236 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.551977 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c192de7b540c08f05f6287097bda1" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:45.552500 13309 raft_consensus.cc:3032] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:45.552708 13309 raft_consensus.cc:729] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:45.553324 13309 consensus_queue.cc:260] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.554111 13309 raft_consensus.cc:3037] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:45.558159 13236 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 4.
I20240503 02:06:45.559232 13309 raft_consensus.cc:2454] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 4.
I20240503 02:06:45.559199 13127 leader_election.cc:302] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ed2caa226f4e12ac39799a6e80e878, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:45.560137 13447 raft_consensus.cc:2781] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:45.560510 13447 raft_consensus.cc:686] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:45.561213 13447 consensus_queue.cc:237] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.567081 13055 catalog_manager.cc:5484] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 3 to 4, leader changed from 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 4 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:45.700457 13471 raft_consensus.cc:979] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: : Instructing follower d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.700892 13478 raft_consensus.cc:1067] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Signalling peer d6ed2caa226f4e12ac39799a6e80e878 to start an election
I20240503 02:06:45.702201 13161 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
from {username='slave'} at 127.0.0.1:60752
I20240503 02:06:45.702735 13161 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:45.703065 13161 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.709395 13161 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.711172 13161 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 3 election: Requested vote from peers ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:45.711992 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:45.712141 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "d6ed2caa226f4e12ac39799a6e80e878" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:45.712594 13236 raft_consensus.cc:3032] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:45.712680 13309 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.712913 13236 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:45.713610 13236 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.714553 13236 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:45.717495 13309 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 3.
I20240503 02:06:45.718483 13125 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, d6ed2caa226f4e12ac39799a6e80e878; no voters:
I20240503 02:06:45.719266 13447 raft_consensus.cc:2781] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:45.719292 13236 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ed2caa226f4e12ac39799a6e80e878 in term 3.
I20240503 02:06:45.719702 13447 raft_consensus.cc:686] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Becoming Leader. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:45.720450 13447 consensus_queue.cc:237] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:45.726126 13055 catalog_manager.cc:5484] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 reported cstate change: term changed from 2 to 3, leader changed from ef52048cb739452e9b4b2266943ec12f (127.12.53.2) to d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1). New cstate: current_term: 3 leader_uuid: "d6ed2caa226f4e12ac39799a6e80e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:45.953186 13309 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:45.954245 13447 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:45.956563 13236 raft_consensus.cc:1261] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:45.957819 13476 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:45.968847 13236 raft_consensus.cc:1261] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:45.969105 13309 raft_consensus.cc:1261] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:45.970407 13476 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:45.976096 13447 consensus_queue.cc:1035] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:46.033844 13306 raft_consensus.cc:1261] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:46.035032 13447 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:46.043607 13234 raft_consensus.cc:1261] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:46.045755 13493 consensus_queue.cc:1035] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:46.066234 13306 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:46.067327 13484 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:46.075194 13234 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:46.076606 13484 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:46.119386 13234 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:46.120498 13484 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:46.128109 13306 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Refusing update from remote peer d6ed2caa226f4e12ac39799a6e80e878: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:46.129494 13476 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:48.426841 12500 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240503 02:06:48.427402 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:48.434136 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:48.434597 13161 raft_consensus.cc:595] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Received request to transfer leadership to TS 2f429bde36194fccbe8624caaa3b6b86
I20240503 02:06:48.436641 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:48.437048 13161 raft_consensus.cc:595] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Received request to transfer leadership to TS ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:48.438717 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:48.439100 13161 raft_consensus.cc:595] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Received request to transfer leadership to TS 2f429bde36194fccbe8624caaa3b6b86
I20240503 02:06:48.440840 13161 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592"
dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
mode: GRACEFUL
new_leader_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:60778
I20240503 02:06:48.441246 13161 raft_consensus.cc:595] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Received request to transfer leadership to TS ef52048cb739452e9b4b2266943ec12f
I20240503 02:06:48.442041 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 02:06:48.442481 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:48.619967 13504 raft_consensus.cc:979] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:48.620427 13505 raft_consensus.cc:1067] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Signalling peer ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:48.621762 13505 raft_consensus.cc:979] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:48.621773 13234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:45334
I20240503 02:06:48.622296 13506 raft_consensus.cc:1067] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Signalling peer ef52048cb739452e9b4b2266943ec12f to start an election
I20240503 02:06:48.622403 13234 raft_consensus.cc:482] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:48.622730 13234 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:48.623633 13236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551"
dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
from {username='slave'} at 127.0.0.1:45334
I20240503 02:06:48.624171 13236 raft_consensus.cc:482] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:48.624519 13236 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:48.628865 13234 raft_consensus.cc:504] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.630576 13236 raft_consensus.cc:504] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.631364 13234 leader_election.cc:288] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 3 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:48.631873 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:48.632419 13161 raft_consensus.cc:3032] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:48.632544 13236 leader_election.cc:288] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 5 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3:46479)
I20240503 02:06:48.633175 13307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7967a694f1614ee3bd3e0bc2e558a592" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:48.633412 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
I20240503 02:06:48.633725 13307 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:48.633903 13306 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:48.636850 13161 raft_consensus.cc:729] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:48.637589 13161 consensus_queue.cc:260] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.638096 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2d90103c624a74b9311910a8d06551" candidate_uuid: "ef52048cb739452e9b4b2266943ec12f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:48.638640 13160 raft_consensus.cc:3032] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:06:48.638733 13161 raft_consensus.cc:3037] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:48.638947 13160 raft_consensus.cc:729] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:48.639837 13160 consensus_queue.cc:260] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.640412 13307 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 3.
I20240503 02:06:48.640959 13160 raft_consensus.cc:3037] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:06:48.640930 13306 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 5.
I20240503 02:06:48.642200 13200 leader_election.cc:302] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:48.643115 13200 leader_election.cc:302] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:48.643985 13478 raft_consensus.cc:2781] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:48.644923 13161 raft_consensus.cc:2454] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 3.
I20240503 02:06:48.645664 13478 raft_consensus.cc:686] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 3 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:48.645979 13508 raft_consensus.cc:2781] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:06:48.646494 13478 consensus_queue.cc:237] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.647871 13505 raft_consensus.cc:979] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:48.648344 13505 raft_consensus.cc:1067] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Signalling peer 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:48.649835 13160 raft_consensus.cc:2454] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef52048cb739452e9b4b2266943ec12f in term 5.
I20240503 02:06:48.649902 13306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:57252
I20240503 02:06:48.650310 13508 raft_consensus.cc:686] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 5 LEADER]: Becoming Leader. State: Replica: ef52048cb739452e9b4b2266943ec12f, State: Running, Role: LEADER
I20240503 02:06:48.650681 13306 raft_consensus.cc:482] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:48.651033 13306 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:48.651391 13508 consensus_queue.cc:237] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.656522 13306 raft_consensus.cc:504] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.656265 13056 catalog_manager.cc:5484] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 2 to 3, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 3 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:48.658769 13306 leader_election.cc:288] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 2 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:48.659786 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:48.660445 13236 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:48.661365 13160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377a299c253f4d6fab023232d6db92ef" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:48.661993 13160 raft_consensus.cc:3032] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:06:48.662302 13160 raft_consensus.cc:729] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:48.663130 13160 consensus_queue.cc:260] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.664283 13160 raft_consensus.cc:3037] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:48.667171 13236 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 2.
I20240503 02:06:48.668514 13276 leader_election.cc:302] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:48.669327 13477 raft_consensus.cc:2781] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:48.670378 13477 raft_consensus.cc:686] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 2 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:48.671305 13477 consensus_queue.cc:237] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:48.672590 13160 raft_consensus.cc:2454] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 2.
I20240503 02:06:48.676548 13056 catalog_manager.cc:5484] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f reported cstate change: term changed from 4 to 5, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to ef52048cb739452e9b4b2266943ec12f (127.12.53.2). New cstate: current_term: 5 leader_uuid: "ef52048cb739452e9b4b2266943ec12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:48.678035 13055 catalog_manager.cc:5484] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 1 to 2, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 2 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:49.033317 13504 raft_consensus.cc:979] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: : Instructing follower 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:49.033732 13506 raft_consensus.cc:1067] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Signalling peer 2f429bde36194fccbe8624caaa3b6b86 to start an election
I20240503 02:06:49.035012 13306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923"
dest_uuid: "2f429bde36194fccbe8624caaa3b6b86"
from {username='slave'} at 127.0.0.1:57252
I20240503 02:06:49.035429 13306 raft_consensus.cc:482] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:06:49.035667 13306 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:49.039651 13306 raft_consensus.cc:504] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:49.040988 13306 leader_election.cc:288] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 4 election: Requested vote from peers d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915), ef52048cb739452e9b4b2266943ec12f (127.12.53.2:38099)
I20240503 02:06:49.041740 13161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d6ed2caa226f4e12ac39799a6e80e878"
I20240503 02:06:49.041882 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4d81de850f4f07a21ff56e03ec3923" candidate_uuid: "2f429bde36194fccbe8624caaa3b6b86" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ef52048cb739452e9b4b2266943ec12f"
I20240503 02:06:49.042227 13161 raft_consensus.cc:3032] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:06:49.042416 13236 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:49.042500 13161 raft_consensus.cc:729] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6ed2caa226f4e12ac39799a6e80e878, State: Running, Role: LEADER
I20240503 02:06:49.043171 13161 consensus_queue.cc:260] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:49.043953 13161 raft_consensus.cc:3037] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:06:49.047219 13236 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 4.
I20240503 02:06:49.048125 13276 leader_election.cc:302] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f429bde36194fccbe8624caaa3b6b86, ef52048cb739452e9b4b2266943ec12f; no voters:
I20240503 02:06:49.048939 13477 raft_consensus.cc:2781] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:06:49.049398 13477 raft_consensus.cc:686] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 LEADER]: Becoming Leader. State: Replica: 2f429bde36194fccbe8624caaa3b6b86, State: Running, Role: LEADER
I20240503 02:06:49.050208 13161 raft_consensus.cc:2454] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f429bde36194fccbe8624caaa3b6b86 in term 4.
I20240503 02:06:49.050738 13477 consensus_queue.cc:237] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } }
I20240503 02:06:49.058107 13055 catalog_manager.cc:5484] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 reported cstate change: term changed from 3 to 4, leader changed from d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1) to 2f429bde36194fccbe8624caaa3b6b86 (127.12.53.3). New cstate: current_term: 4 leader_uuid: "2f429bde36194fccbe8624caaa3b6b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:49.080435 13306 raft_consensus.cc:1261] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:49.081573 13508 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:49.090258 13161 raft_consensus.cc:1261] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 02:06:49.092576 13508 consensus_queue.cc:1035] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:49.173025 13306 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 5 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 02:06:49.174042 13478 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f429bde36194fccbe8624caaa3b6b86" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:06:49.182201 13161 raft_consensus.cc:1261] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 5 FOLLOWER]: Refusing update from remote peer ef52048cb739452e9b4b2266943ec12f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 02:06:49.183463 13508 consensus_queue.cc:1035] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:06:49.185600 13161 raft_consensus.cc:1261] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:49.187275 13516 consensus_queue.cc:1035] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:49.189502 13236 raft_consensus.cc:1261] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:49.190815 13516 consensus_queue.cc:1035] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:49.443256 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:49.448346 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:49.448726 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:49.544545 13161 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:49.545637 13528 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ed2caa226f4e12ac39799a6e80e878" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:49.555191 13236 raft_consensus.cc:1261] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Refusing update from remote peer 2f429bde36194fccbe8624caaa3b6b86: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 02:06:49.556306 13516 consensus_queue.cc:1035] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef52048cb739452e9b4b2266943ec12f" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:50.449666 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:50.455519 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:50.455941 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:51.456701 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:51.461987 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:51.462376 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:52.463081 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:52.468158 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:52.468600 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:53.469431 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:53.474684 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:53.475029 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:54.475802 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:54.481405 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:54.481765 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:55.482436 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:55.487510 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:55.487875 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:56.488615 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:56.493839 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:56.494215 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:57.494989 12500 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:06:57.500386 12500 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:06:57.500833 12500 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:06:58.524094 12500 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240503 02:06:58.524577 12500 tablet_server.cc:178] TabletServer@127.12.53.1:0 shutting down...
I20240503 02:06:58.549134 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:58.549760 12500 tablet_replica.cc:330] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.550717 12500 raft_consensus.cc:2229] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.551461 12500 raft_consensus.cc:2258] T 7967a694f1614ee3bd3e0bc2e558a592 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.553742 12500 tablet_replica.cc:330] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.554255 12500 raft_consensus.cc:2229] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.555238 12500 raft_consensus.cc:2258] T 887ec09cc01447a3912d5fa631258428 P d6ed2caa226f4e12ac39799a6e80e878 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.557003 12500 tablet_replica.cc:330] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.557467 12500 raft_consensus.cc:2229] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.557813 12500 raft_consensus.cc:2258] T 4b2d90103c624a74b9311910a8d06551 P d6ed2caa226f4e12ac39799a6e80e878 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.559717 12500 tablet_replica.cc:330] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
W20240503 02:06:58.559875 13202 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20240503 02:06:58.560222 12500 raft_consensus.cc:2229] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.561261 12500 raft_consensus.cc:2258] T 8395e11e913d4de29d3cd07692664a3c P d6ed2caa226f4e12ac39799a6e80e878 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:06:58.562916 13202 consensus_peers.cc:485] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f -> Peer d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915): Couldn't send request to peer d6ed2caa226f4e12ac39799a6e80e878. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:58.563251 12500 tablet_replica.cc:330] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.563755 12500 raft_consensus.cc:2229] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.564456 12500 raft_consensus.cc:2258] T 096c192de7b540c08f05f6287097bda1 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.566056 12500 tablet_replica.cc:330] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.566521 12500 raft_consensus.cc:2229] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.566885 12500 raft_consensus.cc:2258] T 5f4d81de850f4f07a21ff56e03ec3923 P d6ed2caa226f4e12ac39799a6e80e878 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.568382 12500 tablet_replica.cc:330] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.568924 12500 raft_consensus.cc:2229] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.569423 12500 raft_consensus.cc:2258] T 377a299c253f4d6fab023232d6db92ef P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.571197 12500 tablet_replica.cc:330] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.571679 12500 raft_consensus.cc:2229] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.572079 12500 raft_consensus.cc:2258] T 17b593d939484822907a1512922a5759 P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.573592 12500 tablet_replica.cc:330] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.573994 12500 raft_consensus.cc:2229] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.574335 12500 raft_consensus.cc:2258] T ff10d87bcde049a1b122f1364316acdb P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.575820 12500 tablet_replica.cc:330] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878: stopping tablet replica
I20240503 02:06:58.576213 12500 raft_consensus.cc:2229] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.576937 12500 raft_consensus.cc:2258] T 28648c3d8cc64d0689d0aa50d4a9709d P d6ed2caa226f4e12ac39799a6e80e878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.605930 12500 tablet_server.cc:194] TabletServer@127.12.53.1:0 shutdown complete.
I20240503 02:06:58.624804 12500 tablet_server.cc:178] TabletServer@127.12.53.2:0 shutting down...
W20240503 02:06:58.643357 13202 consensus_peers.cc:485] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f -> Peer d6ed2caa226f4e12ac39799a6e80e878 (127.12.53.1:43915): Couldn't send request to peer d6ed2caa226f4e12ac39799a6e80e878. Status: Network error: Client connection negotiation failed: client connection to 127.12.53.1:43915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:58.650933 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:58.651609 12500 tablet_replica.cc:330] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.652274 12500 raft_consensus.cc:2229] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.652719 12500 raft_consensus.cc:2258] T 377a299c253f4d6fab023232d6db92ef P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.654446 12500 tablet_replica.cc:330] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.654945 12500 raft_consensus.cc:2229] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.655339 12500 raft_consensus.cc:2258] T 887ec09cc01447a3912d5fa631258428 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.656993 12500 tablet_replica.cc:330] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.657481 12500 raft_consensus.cc:2229] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.657898 12500 raft_consensus.cc:2258] T 5f4d81de850f4f07a21ff56e03ec3923 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.659539 12500 tablet_replica.cc:330] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.660226 12500 raft_consensus.cc:2229] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 5 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.661068 12500 raft_consensus.cc:2258] T 4b2d90103c624a74b9311910a8d06551 P ef52048cb739452e9b4b2266943ec12f [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.662575 12500 tablet_replica.cc:330] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.663060 12500 raft_consensus.cc:2229] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.663443 12500 raft_consensus.cc:2258] T 8395e11e913d4de29d3cd07692664a3c P ef52048cb739452e9b4b2266943ec12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.665023 12500 tablet_replica.cc:330] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.665464 12500 raft_consensus.cc:2229] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.666172 12500 raft_consensus.cc:2258] T 7967a694f1614ee3bd3e0bc2e558a592 P ef52048cb739452e9b4b2266943ec12f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.667726 12500 tablet_replica.cc:330] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.668217 12500 raft_consensus.cc:2229] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.668625 12500 raft_consensus.cc:2258] T 17b593d939484822907a1512922a5759 P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.670217 12500 tablet_replica.cc:330] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.670673 12500 raft_consensus.cc:2229] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.671553 12500 raft_consensus.cc:2258] T ff10d87bcde049a1b122f1364316acdb P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.673355 12500 tablet_replica.cc:330] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.673804 12500 raft_consensus.cc:2229] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.674187 12500 raft_consensus.cc:2258] T 096c192de7b540c08f05f6287097bda1 P ef52048cb739452e9b4b2266943ec12f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.676071 12500 tablet_replica.cc:330] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f: stopping tablet replica
I20240503 02:06:58.676499 12500 raft_consensus.cc:2229] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.676959 12500 raft_consensus.cc:2258] T 28648c3d8cc64d0689d0aa50d4a9709d P ef52048cb739452e9b4b2266943ec12f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.704895 12500 tablet_server.cc:194] TabletServer@127.12.53.2:0 shutdown complete.
I20240503 02:06:58.722447 12500 tablet_server.cc:178] TabletServer@127.12.53.3:0 shutting down...
I20240503 02:06:58.744108 12500 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:58.744858 12500 tablet_replica.cc:330] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.745440 12500 raft_consensus.cc:2229] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.745877 12500 raft_consensus.cc:2258] T 7967a694f1614ee3bd3e0bc2e558a592 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.747531 12500 tablet_replica.cc:330] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.748028 12500 raft_consensus.cc:2229] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.748430 12500 raft_consensus.cc:2258] T 096c192de7b540c08f05f6287097bda1 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.750062 12500 tablet_replica.cc:330] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.750608 12500 raft_consensus.cc:2229] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.751387 12500 raft_consensus.cc:2258] T 377a299c253f4d6fab023232d6db92ef P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.753103 12500 tablet_replica.cc:330] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.753545 12500 raft_consensus.cc:2229] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.753954 12500 raft_consensus.cc:2258] T 4b2d90103c624a74b9311910a8d06551 P 2f429bde36194fccbe8624caaa3b6b86 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.755438 12500 tablet_replica.cc:330] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.755867 12500 raft_consensus.cc:2229] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.756222 12500 raft_consensus.cc:2258] T 8395e11e913d4de29d3cd07692664a3c P 2f429bde36194fccbe8624caaa3b6b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.757786 12500 tablet_replica.cc:330] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.758232 12500 raft_consensus.cc:2229] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.758659 12500 raft_consensus.cc:2258] T 887ec09cc01447a3912d5fa631258428 P 2f429bde36194fccbe8624caaa3b6b86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.760288 12500 tablet_replica.cc:330] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.760797 12500 raft_consensus.cc:2229] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.761168 12500 raft_consensus.cc:2258] T 28648c3d8cc64d0689d0aa50d4a9709d P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.762601 12500 tablet_replica.cc:330] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.763021 12500 raft_consensus.cc:2229] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:58.763391 12500 raft_consensus.cc:2258] T ff10d87bcde049a1b122f1364316acdb P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.764947 12500 tablet_replica.cc:330] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.765339 12500 raft_consensus.cc:2229] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.766005 12500 raft_consensus.cc:2258] T 17b593d939484822907a1512922a5759 P 2f429bde36194fccbe8624caaa3b6b86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.767463 12500 tablet_replica.cc:330] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86: stopping tablet replica
I20240503 02:06:58.767880 12500 raft_consensus.cc:2229] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.768496 12500 raft_consensus.cc:2258] T 5f4d81de850f4f07a21ff56e03ec3923 P 2f429bde36194fccbe8624caaa3b6b86 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.794708 12500 tablet_server.cc:194] TabletServer@127.12.53.3:0 shutdown complete.
I20240503 02:06:58.810925 12500 master.cc:537] Master@127.12.53.62:40893 shutting down...
I20240503 02:06:58.831177 12500 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:58.831674 12500 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:58.831940 12500 tablet_replica.cc:330] T 00000000000000000000000000000000 P ef0aa78c99c44b559a6cb65523d50423: stopping tablet replica
I20240503 02:06:58.850085 12500 master.cc:552] Master@127.12.53.62:40893 shutdown complete.
2f429bde36194fccbe8624caaa3b6b86 1
d6ed2caa226f4e12ac39799a6e80e878 5
ef52048cb739452e9b4b2266943ec12f 4
2f429bde36194fccbe8624caaa3b6b86 3
d6ed2caa226f4e12ac39799a6e80e878 3
ef52048cb739452e9b4b2266943ec12f 4
2f429bde36194fccbe8624caaa3b6b86 1
d6ed2caa226f4e12ac39799a6e80e878 8
ef52048cb739452e9b4b2266943ec12f 1
2f429bde36194fccbe8624caaa3b6b86 3
d6ed2caa226f4e12ac39799a6e80e878 4
ef52048cb739452e9b4b2266943ec12f 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28440 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240503 02:06:58.887140 12500 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.53.62:37853
I20240503 02:06:58.888165 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:58.893775 13542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:58.894414 13543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:58.895526 12500 server_base.cc:1034] running on GCE node
W20240503 02:06:58.895547 13545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:58.896571 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:58.896826 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:58.896961 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714702018896943 us; error 0 us; skew 500 ppm
I20240503 02:06:58.897445 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:58.905061 12500 webserver.cc:460] Webserver started at http://127.12.53.62:41369/ using document root <none> and password file <none>
I20240503 02:06:58.905527 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:58.905702 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:58.905975 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:58.907125 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/master-0-root/instance:
uuid: "66bcb220f62f465f8a9c0c95e71424b8"
format_stamp: "Formatted at 2024-05-03 02:06:58 on dist-test-slave-fmt4"
I20240503 02:06:58.911531 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:06:58.914744 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:58.915434 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:06:58.915699 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/master-0-root
uuid: "66bcb220f62f465f8a9c0c95e71424b8"
format_stamp: "Formatted at 2024-05-03 02:06:58 on dist-test-slave-fmt4"
I20240503 02:06:58.916095 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:58.956431 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:58.957693 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:58.994252 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.62:37853
I20240503 02:06:58.994340 13601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.62:37853 every 8 connection(s)
I20240503 02:06:58.997888 13602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.007134 13602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8: Bootstrap starting.
I20240503 02:06:59.011214 13602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:59.015112 13602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8: No bootstrap required, opened a new log
I20240503 02:06:59.017160 13602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER }
I20240503 02:06:59.017692 13602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:59.017894 13602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bcb220f62f465f8a9c0c95e71424b8, State: Initialized, Role: FOLLOWER
I20240503 02:06:59.018460 13602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER }
I20240503 02:06:59.018913 13602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:06:59.019130 13602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:06:59.019380 13602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:59.023216 13602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER }
I20240503 02:06:59.023690 13602 leader_election.cc:302] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bcb220f62f465f8a9c0c95e71424b8; no voters:
I20240503 02:06:59.024734 13602 leader_election.cc:288] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:06:59.025115 13605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:59.026355 13605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [term 1 LEADER]: Becoming Leader. State: Replica: 66bcb220f62f465f8a9c0c95e71424b8, State: Running, Role: LEADER
I20240503 02:06:59.026935 13605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER }
I20240503 02:06:59.027500 13602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:06:59.031021 13606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bcb220f62f465f8a9c0c95e71424b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER } }
I20240503 02:06:59.031194 13607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bcb220f62f465f8a9c0c95e71424b8. Latest consensus state: current_term: 1 leader_uuid: "66bcb220f62f465f8a9c0c95e71424b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bcb220f62f465f8a9c0c95e71424b8" member_type: VOTER } }
I20240503 02:06:59.031584 13606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:59.031641 13607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:59.034718 13611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:06:59.042236 13611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:06:59.046247 12500 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:06:59.054169 13611 catalog_manager.cc:1347] Generated new cluster ID: 794803fa22f34896bd827f43e7480de6
I20240503 02:06:59.054412 13611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:06:59.068210 13611 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:06:59.069384 13611 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:06:59.096328 13611 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66bcb220f62f465f8a9c0c95e71424b8: Generated new TSK 0
I20240503 02:06:59.096927 13611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:06:59.112692 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:59.118606 13623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:59.120685 13624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:59.121018 12500 server_base.cc:1034] running on GCE node
W20240503 02:06:59.121452 13626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:59.122198 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:59.122397 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:59.122545 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714702019122526 us; error 0 us; skew 500 ppm
I20240503 02:06:59.123052 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:59.125259 12500 webserver.cc:460] Webserver started at http://127.12.53.1:41337/ using document root <none> and password file <none>
I20240503 02:06:59.125700 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:59.125880 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:59.126125 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:59.127173 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/instance:
uuid: "7da64b3eda0a40b6b183c1cfc19db738"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.131439 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:06:59.134532 13631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.135201 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:06:59.135459 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root
uuid: "7da64b3eda0a40b6b183c1cfc19db738"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.135740 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:59.147747 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:59.148891 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:59.150221 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:59.152307 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:59.152508 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.152724 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:59.152913 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 02:06:59.189308 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.1:33671
I20240503 02:06:59.189404 13693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.1:33671 every 8 connection(s)
I20240503 02:06:59.193578 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:59.200670 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:59.202739 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:59.204187 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:59.205245 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:59.205621 13694 heartbeater.cc:347] Connected to a master server at 127.12.53.62:37853
I20240503 02:06:59.205957 13694 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:59.206593 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:06:59.206642 13694 heartbeater.cc:511] Master 127.12.53.62:37853 requested a full tablet report, sending...
W20240503 02:06:59.206844 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:59.207068 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714702019207049 us; error 0 us; skew 500 ppm
I20240503 02:06:59.207780 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:59.208914 13567 ts_manager.cc:194] Registered new tserver with Master: 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671)
I20240503 02:06:59.210561 12500 webserver.cc:460] Webserver started at http://127.12.53.2:44453/ using document root <none> and password file <none>
I20240503 02:06:59.211190 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:59.211303 13567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43416
I20240503 02:06:59.211436 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:59.211838 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:59.213132 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/instance:
uuid: "65b596345545434b9f09935907294192"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.217377 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:06:59.220433 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.221159 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:59.221407 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root
uuid: "65b596345545434b9f09935907294192"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.221644 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:59.239763 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:59.241016 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:59.242347 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:59.244422 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:59.244601 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.244851 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:59.244993 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.281625 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.2:42457
I20240503 02:06:59.281733 13768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.2:42457 every 8 connection(s)
I20240503 02:06:59.286003 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:59.293138 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:59.294040 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:59.297093 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:59.297622 13769 heartbeater.cc:347] Connected to a master server at 127.12.53.62:37853
I20240503 02:06:59.298055 13769 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:59.298113 12500 server_base.cc:1034] running on GCE node
I20240503 02:06:59.298899 13769 heartbeater.cc:511] Master 127.12.53.62:37853 requested a full tablet report, sending...
I20240503 02:06:59.298949 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:59.299221 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:59.299388 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714702019299371 us; error 0 us; skew 500 ppm
I20240503 02:06:59.299927 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:59.300701 13567 ts_manager.cc:194] Registered new tserver with Master: 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:06:59.302217 13567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43428
I20240503 02:06:59.302382 12500 webserver.cc:460] Webserver started at http://127.12.53.3:42629/ using document root <none> and password file <none>
I20240503 02:06:59.302886 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:59.303097 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:59.303400 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:59.304641 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/instance:
uuid: "b518e4e142f54c789a56606b766c4e06"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.308791 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:06:59.311777 13780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.312477 12500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:06:59.312765 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root
uuid: "b518e4e142f54c789a56606b766c4e06"
format_stamp: "Formatted at 2024-05-03 02:06:59 on dist-test-slave-fmt4"
I20240503 02:06:59.313043 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:59.352046 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:59.353251 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:59.354665 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:59.356945 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:59.357200 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.357447 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:59.357601 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:59.394433 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.3:35919
I20240503 02:06:59.394527 13842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.3:35919 every 8 connection(s)
I20240503 02:06:59.406908 13843 heartbeater.cc:347] Connected to a master server at 127.12.53.62:37853
I20240503 02:06:59.407279 13843 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:59.407958 13843 heartbeater.cc:511] Master 127.12.53.62:37853 requested a full tablet report, sending...
I20240503 02:06:59.409758 13567 ts_manager.cc:194] Registered new tserver with Master: b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:06:59.410514 12500 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013109038s
I20240503 02:06:59.411047 13567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43440
I20240503 02:06:59.412930 12500 test_util.cc:268] Using random seed: 750994351
I20240503 02:06:59.434536 13567 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43444:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:59.440460 13567 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:59.626891 13734 tablet_service.cc:1467] Processing CreateTablet for tablet 330da4b1ff064e5f865ec87c4b081cce (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 02:06:59.628361 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330da4b1ff064e5f865ec87c4b081cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.629412 13728 tablet_service.cc:1467] Processing CreateTablet for tablet c782825ce834414baad8c3f6b6642e5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 02:06:59.629876 13733 tablet_service.cc:1467] Processing CreateTablet for tablet a8e8e9e6dec94158989853c4dbb2787b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 02:06:59.630757 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c782825ce834414baad8c3f6b6642e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.631495 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e8e9e6dec94158989853c4dbb2787b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.631640 13727 tablet_service.cc:1467] Processing CreateTablet for tablet a24625c57a084e26a113238421e92823 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 02:06:59.632969 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24625c57a084e26a113238421e92823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.632771 13729 tablet_service.cc:1467] Processing CreateTablet for tablet 707d1ec02f89489092029bf2eee1761d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 02:06:59.636459 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ec7430d7f43baadcc6dd1296d7b28 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 02:06:59.637761 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ec7430d7f43baadcc6dd1296d7b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.637487 13726 tablet_service.cc:1467] Processing CreateTablet for tablet c4f6dafbb5974de589d4064a5f979f01 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 02:06:59.638715 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f6dafbb5974de589d4064a5f979f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.639777 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed43100e614122b8bbc0bc3506d664 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 02:06:59.641000 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed43100e614122b8bbc0bc3506d664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.641839 13725 tablet_service.cc:1467] Processing CreateTablet for tablet c4a20af2756040cea6c15f6ace3912bb (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 02:06:59.643026 13725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a20af2756040cea6c15f6ace3912bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.645834 13730 tablet_service.cc:1467] Processing CreateTablet for tablet fe9c8e98bdb04950a65590a69bf0ac19 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 02:06:59.647060 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9c8e98bdb04950a65590a69bf0ac19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.653682 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 707d1ec02f89489092029bf2eee1761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.681826 13807 tablet_service.cc:1467] Processing CreateTablet for tablet a8e8e9e6dec94158989853c4dbb2787b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 02:06:59.683199 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e8e9e6dec94158989853c4dbb2787b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.685901 13806 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ec7430d7f43baadcc6dd1296d7b28 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 02:06:59.687175 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ec7430d7f43baadcc6dd1296d7b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.689731 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed43100e614122b8bbc0bc3506d664 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 02:06:59.690995 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed43100e614122b8bbc0bc3506d664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.693835 13804 tablet_service.cc:1467] Processing CreateTablet for tablet fe9c8e98bdb04950a65590a69bf0ac19 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 02:06:59.694569 13734 tablet_service.cc:1467] Processing CreateTablet for tablet e893a056b25c462d82b4147b937c8c94 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 02:06:59.695217 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9c8e98bdb04950a65590a69bf0ac19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.697877 13802 tablet_service.cc:1467] Processing CreateTablet for tablet c782825ce834414baad8c3f6b6642e5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 02:06:59.699128 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c782825ce834414baad8c3f6b6642e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.699981 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 330da4b1ff064e5f865ec87c4b081cce (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 02:06:59.701233 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330da4b1ff064e5f865ec87c4b081cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.702538 13800 tablet_service.cc:1467] Processing CreateTablet for tablet c4f6dafbb5974de589d4064a5f979f01 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 02:06:59.703807 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f6dafbb5974de589d4064a5f979f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.703950 13659 tablet_service.cc:1467] Processing CreateTablet for tablet 330da4b1ff064e5f865ec87c4b081cce (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 02:06:59.710565 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330da4b1ff064e5f865ec87c4b081cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.712210 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 707d1ec02f89489092029bf2eee1761d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 02:06:59.712903 13651 tablet_service.cc:1467] Processing CreateTablet for tablet c4f6dafbb5974de589d4064a5f979f01 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 02:06:59.713927 13654 tablet_service.cc:1467] Processing CreateTablet for tablet 707d1ec02f89489092029bf2eee1761d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 02:06:59.715058 13650 tablet_service.cc:1467] Processing CreateTablet for tablet c4a20af2756040cea6c15f6ace3912bb (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 02:06:59.715987 13653 tablet_service.cc:1467] Processing CreateTablet for tablet c782825ce834414baad8c3f6b6642e5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 02:06:59.717566 13652 tablet_service.cc:1467] Processing CreateTablet for tablet a24625c57a084e26a113238421e92823 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 02:06:59.717758 13658 tablet_service.cc:1467] Processing CreateTablet for tablet a8e8e9e6dec94158989853c4dbb2787b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 02:06:59.719251 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ec7430d7f43baadcc6dd1296d7b28 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 02:06:59.719372 13656 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed43100e614122b8bbc0bc3506d664 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 02:06:59.723337 13799 tablet_service.cc:1467] Processing CreateTablet for tablet c4a20af2756040cea6c15f6ace3912bb (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 02:06:59.724083 13801 tablet_service.cc:1467] Processing CreateTablet for tablet a24625c57a084e26a113238421e92823 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 02:06:59.727803 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 707d1ec02f89489092029bf2eee1761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.730397 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a20af2756040cea6c15f6ace3912bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.731101 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24625c57a084e26a113238421e92823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.737112 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 2524d54d0a2b4efc84ca69b6057c97ef (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 02:06:59.739459 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24625c57a084e26a113238421e92823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.740092 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c782825ce834414baad8c3f6b6642e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.730198 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893a056b25c462d82b4147b937c8c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.723295 13655 tablet_service.cc:1467] Processing CreateTablet for tablet fe9c8e98bdb04950a65590a69bf0ac19 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 02:06:59.751506 13733 tablet_service.cc:1467] Processing CreateTablet for tablet c5cf17ea09b444b2afbc0d9a5bd38b79 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 02:06:59.752249 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 02:06:59.752872 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 707d1ec02f89489092029bf2eee1761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.759753 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f6dafbb5974de589d4064a5f979f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.767743 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a20af2756040cea6c15f6ace3912bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.779034 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.779512 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed43100e614122b8bbc0bc3506d664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.779755 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cf17ea09b444b2afbc0d9a5bd38b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.780282 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9c8e98bdb04950a65590a69bf0ac19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.780875 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e8e9e6dec94158989853c4dbb2787b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.785084 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ec7430d7f43baadcc6dd1296d7b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.800279 13727 tablet_service.cc:1467] Processing CreateTablet for tablet 294969cac87345fa9dec0a1ed3b214f6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 02:06:59.822829 13801 tablet_service.cc:1467] Processing CreateTablet for tablet e893a056b25c462d82b4147b937c8c94 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 02:06:59.824682 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893a056b25c462d82b4147b937c8c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.828560 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 60da30fcdc7f4fee9a61dd710d55673c (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 02:06:59.837083 13858 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:06:59.844086 13655 tablet_service.cc:1467] Processing CreateTablet for tablet e893a056b25c462d82b4147b937c8c94 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 02:06:59.846573 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2524d54d0a2b4efc84ca69b6057c97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.813099 13726 tablet_service.cc:1467] Processing CreateTablet for tablet 26b581675616419d8d97c4e8973455f4 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 02:06:59.812911 13725 tablet_service.cc:1467] Processing CreateTablet for tablet ee7c74febdf447aebc8b991b45c4e60a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 02:06:59.852991 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 2493a234a8424ef6a6393a690a9f8f5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 02:06:59.852227 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 294969cac87345fa9dec0a1ed3b214f6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 02:06:59.854488 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294969cac87345fa9dec0a1ed3b214f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.862115 13858 tablet_bootstrap.cc:654] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:59.869359 13802 tablet_service.cc:1467] Processing CreateTablet for tablet ee7c74febdf447aebc8b991b45c4e60a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 02:06:59.870635 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7c74febdf447aebc8b991b45c4e60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.871186 13806 tablet_service.cc:1467] Processing CreateTablet for tablet c5cf17ea09b444b2afbc0d9a5bd38b79 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 02:06:59.872359 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cf17ea09b444b2afbc0d9a5bd38b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.920776 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893a056b25c462d82b4147b937c8c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.930254 13654 tablet_service.cc:1467] Processing CreateTablet for tablet ee7c74febdf447aebc8b991b45c4e60a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 02:06:59.932140 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7c74febdf447aebc8b991b45c4e60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.953495 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2493a234a8424ef6a6393a690a9f8f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.961310 13804 tablet_service.cc:1467] Processing CreateTablet for tablet 26b581675616419d8d97c4e8973455f4 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 02:06:59.963387 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b581675616419d8d97c4e8973455f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:59.963783 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 2524d54d0a2b4efc84ca69b6057c97ef (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 02:06:59.983144 13729 tablet_service.cc:1467] Processing CreateTablet for tablet 7b722a96f2794d91bbd9163a7f283c27 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 02:06:59.985191 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2524d54d0a2b4efc84ca69b6057c97ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:06:59.988058 13553 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 88432.9us
W20240503 02:06:59.991493 13551 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 81331us
I20240503 02:06:59.995188 13861 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:00.001196 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b581675616419d8d97c4e8973455f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.002138 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60da30fcdc7f4fee9a61dd710d55673c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.003454 13800 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 02:07:00.005088 13860 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:00.005157 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294969cac87345fa9dec0a1ed3b214f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.019390 13807 tablet_service.cc:1467] Processing CreateTablet for tablet 7b722a96f2794d91bbd9163a7f283c27 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 02:07:00.007247 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 60da30fcdc7f4fee9a61dd710d55673c (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 02:07:00.022388 13725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7c74febdf447aebc8b991b45c4e60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.024166 13860 tablet_bootstrap.cc:654] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.025957 13733 tablet_service.cc:1467] Processing CreateTablet for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240503 02:07:00.027218 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.009552 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 2493a234a8424ef6a6393a690a9f8f5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 02:07:00.015431 13728 tablet_service.cc:1467] Processing CreateTablet for tablet fb6854ff6c7c458c89a06fc412aca3f0 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 02:07:00.034005 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6854ff6c7c458c89a06fc412aca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.035567 13653 tablet_service.cc:1467] Processing CreateTablet for tablet 294969cac87345fa9dec0a1ed3b214f6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 02:07:00.038717 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294969cac87345fa9dec0a1ed3b214f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.038395 13659 tablet_service.cc:1467] Processing CreateTablet for tablet 26b581675616419d8d97c4e8973455f4 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 02:07:00.039570 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b581675616419d8d97c4e8973455f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:07:00.043154 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:00.044571 13657 tablet_service.cc:1467] Processing CreateTablet for tablet c5cf17ea09b444b2afbc0d9a5bd38b79 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 02:07:00.045817 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cf17ea09b444b2afbc0d9a5bd38b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.049221 13803 tablet_service.cc:1467] Processing CreateTablet for tablet fb6854ff6c7c458c89a06fc412aca3f0 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 02:07:00.053572 13861 tablet_bootstrap.cc:654] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.058312 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b722a96f2794d91bbd9163a7f283c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.061610 13801 tablet_service.cc:1467] Processing CreateTablet for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240503 02:07:00.073146 13858 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:00.073861 13858 ts_tablet_manager.cc:1367] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.264s user 0.016s sys 0.007s
I20240503 02:07:00.075580 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 7d610b8f0ef44e33880de53f13a9dd3e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 02:07:00.020620 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b722a96f2794d91bbd9163a7f283c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.081022 13652 tablet_service.cc:1467] Processing CreateTablet for tablet 2524d54d0a2b4efc84ca69b6057c97ef (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 02:07:00.082270 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2524d54d0a2b4efc84ca69b6057c97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.086228 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 02:07:00.081122 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60da30fcdc7f4fee9a61dd710d55673c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.093165 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.088470 13656 tablet_service.cc:1467] Processing CreateTablet for tablet 60da30fcdc7f4fee9a61dd710d55673c (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 02:07:00.095046 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d610b8f0ef44e33880de53f13a9dd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.099174 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2493a234a8424ef6a6393a690a9f8f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.101632 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60da30fcdc7f4fee9a61dd710d55673c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.105749 13858 raft_consensus.cc:348] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.107087 13858 raft_consensus.cc:374] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.107393 13858 raft_consensus.cc:729] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.108268 13858 consensus_queue.cc:260] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.112849 13861 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:00.115825 13861 ts_tablet_manager.cc:1367] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.000s
I20240503 02:07:00.118643 13861 raft_consensus.cc:348] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.119338 13861 raft_consensus.cc:374] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.119664 13802 tablet_service.cc:1467] Processing CreateTablet for tablet 7d610b8f0ef44e33880de53f13a9dd3e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 02:07:00.123713 13726 tablet_service.cc:1467] Processing CreateTablet for tablet 4904ea1d27d64f01a7ce69d4efa093af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 02:07:00.113245 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.124413 13804 tablet_service.cc:1467] Processing CreateTablet for tablet 07abdb29bc6d463c8928a23e8579ebac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 02:07:00.125483 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2a2e3ecd3f407ca6cd7a32ac82e89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.119637 13861 raft_consensus.cc:729] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.135116 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6854ff6c7c458c89a06fc412aca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.128542 13651 tablet_service.cc:1467] Processing CreateTablet for tablet 2493a234a8424ef6a6393a690a9f8f5d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 02:07:00.147930 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2493a234a8424ef6a6393a690a9f8f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.134541 13654 tablet_service.cc:1467] Processing CreateTablet for tablet fb6854ff6c7c458c89a06fc412aca3f0 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 02:07:00.129370 13734 tablet_service.cc:1467] Processing CreateTablet for tablet 07abdb29bc6d463c8928a23e8579ebac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 02:07:00.137691 13658 tablet_service.cc:1467] Processing CreateTablet for tablet 7b722a96f2794d91bbd9163a7f283c27 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 02:07:00.152731 13861 consensus_queue.cc:260] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.133836 13806 tablet_service.cc:1467] Processing CreateTablet for tablet 4904ea1d27d64f01a7ce69d4efa093af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 02:07:00.127570 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904ea1d27d64f01a7ce69d4efa093af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.159322 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 1251d1d9cfc6460ca25952c2cfa21817 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 02:07:00.160238 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07abdb29bc6d463c8928a23e8579ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.163151 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d610b8f0ef44e33880de53f13a9dd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.165055 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07abdb29bc6d463c8928a23e8579ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.137089 13860 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:00.170748 13860 ts_tablet_manager.cc:1367] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.166s user 0.007s sys 0.012s
I20240503 02:07:00.140466 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 4b89ae3ce565499ebcaab8912fedc385 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 02:07:00.171787 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b89ae3ce565499ebcaab8912fedc385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.157279 13653 tablet_service.cc:1467] Processing CreateTablet for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240503 02:07:00.162182 13725 tablet_service.cc:1467] Processing CreateTablet for tablet b19c9d349b744a759025ccddcf3633a6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 02:07:00.174031 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 7d610b8f0ef44e33880de53f13a9dd3e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 02:07:00.175271 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904ea1d27d64f01a7ce69d4efa093af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.175839 13733 tablet_service.cc:1467] Processing CreateTablet for tablet a71d633c024e481d9dcd7dc94259a65d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 02:07:00.177358 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1251d1d9cfc6460ca25952c2cfa21817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.133513 13727 tablet_service.cc:1467] Processing CreateTablet for tablet 1251d1d9cfc6460ca25952c2cfa21817 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 02:07:00.169580 13769 heartbeater.cc:503] Master 127.12.53.62:37853 was elected leader, sending a full tablet report...
I20240503 02:07:00.149910 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6854ff6c7c458c89a06fc412aca3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.169916 13729 tablet_service.cc:1467] Processing CreateTablet for tablet b15197927046474b804042923de03d31 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 02:07:00.159794 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 78a74eddb0bc474cb7c1e14749665244 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 02:07:00.194907 13858 ts_tablet_manager.cc:1398] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.121s user 0.010s sys 0.021s
I20240503 02:07:00.201321 13725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19c9d349b744a759025ccddcf3633a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.208367 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d610b8f0ef44e33880de53f13a9dd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.211308 13860 raft_consensus.cc:348] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.212097 13860 raft_consensus.cc:374] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.212399 13860 raft_consensus.cc:729] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.213238 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b722a96f2794d91bbd9163a7f283c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.213312 13860 consensus_queue.cc:260] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.214103 13694 heartbeater.cc:503] Master 127.12.53.62:37853 was elected leader, sending a full tablet report...
I20240503 02:07:00.214780 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab629d73c1c84dbbb3c7d3cd00dedf2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.219777 13858 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:00.237149 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a74eddb0bc474cb7c1e14749665244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.245083 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15197927046474b804042923de03d31. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:07:00.249037 13551 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 44863.1us
I20240503 02:07:00.253358 13728 tablet_service.cc:1467] Processing CreateTablet for tablet d85f42f870e749bb9cd2aecf025d3a20 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 02:07:00.257086 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d633c024e481d9dcd7dc94259a65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.264017 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 4b89ae3ce565499ebcaab8912fedc385 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 02:07:00.289323 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b89ae3ce565499ebcaab8912fedc385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.295477 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 1251d1d9cfc6460ca25952c2cfa21817 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 02:07:00.296466 13858 tablet_bootstrap.cc:654] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.273329 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 78a74eddb0bc474cb7c1e14749665244 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 02:07:00.274386 13804 tablet_service.cc:1467] Processing CreateTablet for tablet b19c9d349b744a759025ccddcf3633a6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 02:07:00.276232 13861 ts_tablet_manager.cc:1398] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.160s user 0.009s sys 0.009s
I20240503 02:07:00.278112 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 67b296978fa84ad6a6f9cc7ec6d202ac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
W20240503 02:07:00.286089 13770 tablet.cc:2369] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:07:00.282433 13808 tablet_service.cc:1467] Processing CreateTablet for tablet b15197927046474b804042923de03d31 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 02:07:00.264685 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1251d1d9cfc6460ca25952c2cfa21817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.325713 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a74eddb0bc474cb7c1e14749665244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.292531 13659 tablet_service.cc:1467] Processing CreateTablet for tablet 4904ea1d27d64f01a7ce69d4efa093af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 02:07:00.302099 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 4b89ae3ce565499ebcaab8912fedc385 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 02:07:00.328625 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19c9d349b744a759025ccddcf3633a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.271651 13652 tablet_service.cc:1467] Processing CreateTablet for tablet 07abdb29bc6d463c8928a23e8579ebac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 02:07:00.329382 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15197927046474b804042923de03d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.287556 13800 tablet_service.cc:1467] Processing CreateTablet for tablet a71d633c024e481d9dcd7dc94259a65d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 02:07:00.330425 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d633c024e481d9dcd7dc94259a65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.331401 13861 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:00.310020 13656 tablet_service.cc:1467] Processing CreateTablet for tablet b8461a9dc6344866b9e069d0b28ee64d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 02:07:00.315240 13726 tablet_service.cc:1467] Processing CreateTablet for tablet b1ac22299784411188965b2bfbd51ef6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 02:07:00.317991 13801 tablet_service.cc:1467] Processing CreateTablet for tablet d85f42f870e749bb9cd2aecf025d3a20 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 02:07:00.337132 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85f42f870e749bb9cd2aecf025d3a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.337949 13866 raft_consensus.cc:482] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:00.338424 13866 raft_consensus.cc:504] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.339989 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85f42f870e749bb9cd2aecf025d3a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.343633 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1251d1d9cfc6460ca25952c2cfa21817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.322408 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 67b296978fa84ad6a6f9cc7ec6d202ac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240503 02:07:00.344635 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b89ae3ce565499ebcaab8912fedc385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.346189 13860 ts_tablet_manager.cc:1398] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.175s user 0.034s sys 0.027s
I20240503 02:07:00.353075 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07abdb29bc6d463c8928a23e8579ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.354502 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b296978fa84ad6a6f9cc7ec6d202ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.372224 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ac22299784411188965b2bfbd51ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.374673 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8461a9dc6344866b9e069d0b28ee64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.376034 13861 tablet_bootstrap.cc:654] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.385896 13802 tablet_service.cc:1467] Processing CreateTablet for tablet a0b623dc663b454a8380dad2c487d035 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 02:07:00.359079 13734 tablet_service.cc:1467] Processing CreateTablet for tablet a0b623dc663b454a8380dad2c487d035 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 02:07:00.371034 13843 heartbeater.cc:503] Master 127.12.53.62:37853 was elected leader, sending a full tablet report...
I20240503 02:07:00.395030 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4904ea1d27d64f01a7ce69d4efa093af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.409298 13865 raft_consensus.cc:482] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:00.409832 13865 raft_consensus.cc:504] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.411057 13866 leader_election.cc:288] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:00.416405 13865 leader_election.cc:288] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:00.416096 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 4331632ebc06477a896d103b58707502 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240503 02:07:00.359079 13807 tablet_service.cc:1467] Processing CreateTablet for tablet b1ac22299784411188965b2bfbd51ef6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 02:07:00.431699 13860 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:00.416963 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 4331632ebc06477a896d103b58707502 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240503 02:07:00.430162 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b623dc663b454a8380dad2c487d035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.448115 13860 tablet_bootstrap.cc:654] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.447964 13728 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7f8c910f68477585088d11c8c0f9af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240503 02:07:00.461113 13654 tablet_service.cc:1467] Processing CreateTablet for tablet 78a74eddb0bc474cb7c1e14749665244 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 02:07:00.462675 13861 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:00.463155 13861 ts_tablet_manager.cc:1367] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.019s
I20240503 02:07:00.468128 13858 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:00.468614 13858 ts_tablet_manager.cc:1367] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.249s user 0.012s sys 0.003s
I20240503 02:07:00.468331 13861 raft_consensus.cc:348] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.469050 13861 raft_consensus.cc:374] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.469429 13861 raft_consensus.cc:729] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.470103 13861 consensus_queue.cc:260] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.481433 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4331632ebc06477a896d103b58707502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.500344 13725 tablet_service.cc:1467] Processing CreateTablet for tablet bc74b8e531c942f28a4b9467a14baf26 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 02:07:00.502266 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a74eddb0bc474cb7c1e14749665244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.504107 13806 tablet_service.cc:1467] Processing CreateTablet for tablet bc74b8e531c942f28a4b9467a14baf26 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 02:07:00.522665 13651 tablet_service.cc:1467] Processing CreateTablet for tablet b19c9d349b744a759025ccddcf3633a6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 02:07:00.524106 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19c9d349b744a759025ccddcf3633a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.514707 13727 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7a151bad744d7ab3ed1effcd61e49 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 02:07:00.517165 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:00.526331 13650 tablet_service.cc:1467] Processing CreateTablet for tablet a71d633c024e481d9dcd7dc94259a65d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 02:07:00.527693 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d633c024e481d9dcd7dc94259a65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.527526 13808 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7f8c910f68477585088d11c8c0f9af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
W20240503 02:07:00.528079 13632 leader_election.cc:341] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.530104 13658 tablet_service.cc:1467] Processing CreateTablet for tablet 67b296978fa84ad6a6f9cc7ec6d202ac (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240503 02:07:00.531306 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b296978fa84ad6a6f9cc7ec6d202ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.532806 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b623dc663b454a8380dad2c487d035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.526283 13656 tablet_service.cc:1467] Processing CreateTablet for tablet d85f42f870e749bb9cd2aecf025d3a20 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 02:07:00.533951 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85f42f870e749bb9cd2aecf025d3a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.522665 13653 tablet_service.cc:1467] Processing CreateTablet for tablet b15197927046474b804042923de03d31 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 02:07:00.536219 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15197927046474b804042923de03d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.536995 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b296978fa84ad6a6f9cc7ec6d202ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.541085 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ac22299784411188965b2bfbd51ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.542260 13800 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7a151bad744d7ab3ed1effcd61e49 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 02:07:00.543493 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7a151bad744d7ab3ed1effcd61e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.541139 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7f8c910f68477585088d11c8c0f9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.491138 13858 raft_consensus.cc:348] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.547669 13858 raft_consensus.cc:374] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.547966 13858 raft_consensus.cc:729] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.549499 13858 consensus_queue.cc:260] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.551836 13858 ts_tablet_manager.cc:1398] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.083s user 0.004s sys 0.002s
I20240503 02:07:00.554962 13858 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:00.564966 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4331632ebc06477a896d103b58707502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.569013 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74b8e531c942f28a4b9467a14baf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.570672 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 4459e1c4e013400f920bba94d281d95e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240503 02:07:00.572029 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4459e1c4e013400f920bba94d281d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.584482 13861 ts_tablet_manager.cc:1398] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.121s user 0.005s sys 0.003s
I20240503 02:07:00.585516 13861 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:00.581065 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7a151bad744d7ab3ed1effcd61e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.598511 13804 tablet_service.cc:1467] Processing CreateTablet for tablet c74128b771f146e78210a5b8dcf823a8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 02:07:00.587826 13801 tablet_service.cc:1467] Processing CreateTablet for tablet ee01df95e61d4570a6c7827cd2acc525 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 02:07:00.606189 13732 tablet_service.cc:1467] Processing CreateTablet for tablet c74128b771f146e78210a5b8dcf823a8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 02:07:00.614410 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:00.615356 13861 tablet_bootstrap.cc:654] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.615401 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 97598aefb41c4115a37385063f8d0eb7 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
W20240503 02:07:00.616389 13709 leader_election.cc:341] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.631772 13726 tablet_service.cc:1467] Processing CreateTablet for tablet ee01df95e61d4570a6c7827cd2acc525 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 02:07:00.603427 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7f8c910f68477585088d11c8c0f9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.647583 13733 tablet_service.cc:1467] Processing CreateTablet for tablet 1d086d958d664a1ca6d0a0cc9324f480 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240503 02:07:00.647814 13729 tablet_service.cc:1467] Processing CreateTablet for tablet 4459e1c4e013400f920bba94d281d95e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240503 02:07:00.648572 13652 tablet_service.cc:1467] Processing CreateTablet for tablet b1ac22299784411188965b2bfbd51ef6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 02:07:00.649781 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ac22299784411188965b2bfbd51ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.651288 13860 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:00.651757 13860 ts_tablet_manager.cc:1367] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.220s user 0.004s sys 0.016s
I20240503 02:07:00.652271 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74128b771f146e78210a5b8dcf823a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.654140 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 02:07:00.655305 13860 raft_consensus.cc:348] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.656138 13860 raft_consensus.cc:374] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.635459 13725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74b8e531c942f28a4b9467a14baf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.639297 13867 raft_consensus.cc:482] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:00.656584 13653 tablet_service.cc:1467] Processing CreateTablet for tablet a0b623dc663b454a8380dad2c487d035 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 02:07:00.657053 13867 raft_consensus.cc:504] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.657861 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b623dc663b454a8380dad2c487d035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.661057 13866 raft_consensus.cc:482] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:00.661530 13866 raft_consensus.cc:504] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.663396 13866 leader_election.cc:288] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:00.663375 13806 tablet_service.cc:1467] Processing CreateTablet for tablet 1d086d958d664a1ca6d0a0cc9324f480 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240503 02:07:00.632911 13802 tablet_service.cc:1467] Processing CreateTablet for tablet 97598aefb41c4115a37385063f8d0eb7 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240503 02:07:00.664590 13867 leader_election.cc:288] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:00.671103 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:00.672592 13632 leader_election.cc:341] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.657038 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee01df95e61d4570a6c7827cd2acc525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.656481 13860 raft_consensus.cc:729] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.677209 13860 consensus_queue.cc:260] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.679482 13860 ts_tablet_manager.cc:1398] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20240503 02:07:00.680394 13860 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:00.682276 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7f8c910f68477585088d11c8c0f9af (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240503 02:07:00.683585 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7f8c910f68477585088d11c8c0f9af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.691563 13651 tablet_service.cc:1467] Processing CreateTablet for tablet 4459e1c4e013400f920bba94d281d95e (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240503 02:07:00.693668 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:00.695154 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:00.695253 13728 tablet_service.cc:1467] Processing CreateTablet for tablet b8461a9dc6344866b9e069d0b28ee64d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 02:07:00.660769 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 4331632ebc06477a896d103b58707502 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20240503 02:07:00.697537 13635 leader_election.cc:341] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.706194 13635 leader_election.cc:302] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
W20240503 02:07:00.708539 13635 leader_election.cc:341] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.709060 13635 leader_election.cc:302] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:00.709954 13868 raft_consensus.cc:2726] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:00.710171 13866 raft_consensus.cc:2726] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:00.710763 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4459e1c4e013400f920bba94d281d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.707227 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:00.725230 13650 tablet_service.cc:1467] Processing CreateTablet for tablet ee01df95e61d4570a6c7827cd2acc525 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 02:07:00.725598 13803 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6b185135347d58dfb8ddd0b758e17 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 02:07:00.726855 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97598aefb41c4115a37385063f8d0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.727648 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74128b771f146e78210a5b8dcf823a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.733696 13800 tablet_service.cc:1467] Processing CreateTablet for tablet b8461a9dc6344866b9e069d0b28ee64d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 02:07:00.734432 13804 tablet_service.cc:1467] Processing CreateTablet for tablet b664b6227e0f4268a89935f722d56c8f (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20240503 02:07:00.737876 13710 leader_election.cc:341] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.738580 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4331632ebc06477a896d103b58707502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.738375 13710 leader_election.cc:302] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:00.742333 13861 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:00.742328 13865 raft_consensus.cc:2726] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:00.742960 13861 ts_tablet_manager.cc:1367] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.158s user 0.020s sys 0.008s
I20240503 02:07:00.738162 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d086d958d664a1ca6d0a0cc9324f480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.744678 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee01df95e61d4570a6c7827cd2acc525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.746510 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.747848 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8461a9dc6344866b9e069d0b28ee64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.716018 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97598aefb41c4115a37385063f8d0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.749641 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d086d958d664a1ca6d0a0cc9324f480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.750771 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b664b6227e0f4268a89935f722d56c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.751116 13867 raft_consensus.cc:482] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:00.759550 13867 raft_consensus.cc:504] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.761324 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee01df95e61d4570a6c7827cd2acc525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.706939 13656 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7a151bad744d7ab3ed1effcd61e49 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 02:07:00.705850 13659 tablet_service.cc:1467] Processing CreateTablet for tablet bc74b8e531c942f28a4b9467a14baf26 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 02:07:00.764966 13867 leader_election.cc:288] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:00.753952 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4459e1c4e013400f920bba94d281d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.688537 13807 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 02:07:00.756436 13858 tablet_bootstrap.cc:654] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.765379 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7a151bad744d7ab3ed1effcd61e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.771569 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8461a9dc6344866b9e069d0b28ee64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.778653 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 02:07:00.779844 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.783589 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:00.773214 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:00.769703 13805 tablet_service.cc:1467] Processing CreateTablet for tablet d67aeb523f5b4da696a2135fb1acb234 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 02:07:00.784974 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24625c57a084e26a113238421e92823" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:00.775936 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24625c57a084e26a113238421e92823" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:00.785854 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67aeb523f5b4da696a2135fb1acb234. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:07:00.786319 13781 leader_election.cc:341] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.785785 13743 raft_consensus.cc:2454] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
W20240503 02:07:00.786964 13783 leader_election.cc:341] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.787549 13783 leader_election.cc:302] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:00.787822 13781 leader_election.cc:302] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:00.777853 13861 raft_consensus.cc:348] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.789712 13867 raft_consensus.cc:2726] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:00.790352 13861 raft_consensus.cc:374] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.791960 13861 raft_consensus.cc:729] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:00.794559 13658 tablet_service.cc:1467] Processing CreateTablet for tablet c74128b771f146e78210a5b8dcf823a8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 02:07:00.791920 13867 raft_consensus.cc:2781] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:00.795506 13867 raft_consensus.cc:482] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:00.795833 13867 raft_consensus.cc:3037] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:00.797062 13861 consensus_queue.cc:260] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.798974 13654 tablet_service.cc:1467] Processing CreateTablet for tablet 97598aefb41c4115a37385063f8d0eb7 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240503 02:07:00.801326 13734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 34) took 1070 ms. Trace:
I20240503 02:07:00.801744 13734 rpcz_store.cc:276] 0503 02:06:59.731408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.358113 (+626705us) service_pool.cc:225] Handling call
0503 02:07:00.394679 (+ 36566us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.401114 (+ 6435us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.680135 (+279021us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.800854 (+120719us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.801270 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.803934 13734 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6b185135347d58dfb8ddd0b758e17 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 02:07:00.807092 13861 ts_tablet_manager.cc:1398] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.064s user 0.008s sys 0.000s
I20240503 02:07:00.808089 13861 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:00.816916 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.820670 13808 tablet_service.cc:1467] Processing CreateTablet for tablet a321281efc6f4adf8d7fc972c85eef37 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 02:07:00.821502 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6b185135347d58dfb8ddd0b758e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.822121 13808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a321281efc6f4adf8d7fc972c85eef37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.822592 13733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 43) took 1020 ms. Trace:
W20240503 02:07:00.822809 13783 leader_election.cc:341] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.830950 13733 rpcz_store.cc:276] 0503 02:06:59.802068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.616827 (+814759us) service_pool.cc:225] Handling call
0503 02:07:00.648444 (+ 31617us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.648593 (+ 149us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.779601 (+131008us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.822148 (+ 42547us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.822534 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19200}
I20240503 02:07:00.833896 13733 tablet_service.cc:1467] Processing CreateTablet for tablet b664b6227e0f4268a89935f722d56c8f (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240503 02:07:00.834430 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74b8e531c942f28a4b9467a14baf26. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:07:00.835322 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 47569.9us
I20240503 02:07:00.841852 13652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 33) took 1013 ms. Trace:
I20240503 02:07:00.842235 13652 rpcz_store.cc:276] 0503 02:06:59.828061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.647717 (+819656us) service_pool.cc:225] Handling call
0503 02:07:00.649364 (+ 1647us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.649454 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.776040 (+126586us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.840011 (+ 63971us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.841796 (+ 1785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.841287 13730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 42) took 1051 ms. Trace:
I20240503 02:07:00.844022 13730 rpcz_store.cc:276] 0503 02:06:59.790018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.614409 (+824391us) service_pool.cc:225] Handling call
0503 02:07:00.616306 (+ 1897us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.616412 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.796233 (+179821us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.836628 (+ 40395us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.840581 (+ 3953us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.846961 13800 tablet_service.cc:1467] Processing CreateTablet for tablet a2243e6f9b764eda92cfa44cb769fde6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 02:07:00.848222 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2243e6f9b764eda92cfa44cb769fde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.849906 13729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 39) took 1086 ms. Trace:
I20240503 02:07:00.850783 13732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 41) took 1081 ms. Trace:
I20240503 02:07:00.851181 13732 rpcz_store.cc:276] 0503 02:06:59.769503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.605213 (+835710us) service_pool.cc:225] Handling call
0503 02:07:00.606995 (+ 1782us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.607087 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.798714 (+191627us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.850336 (+ 51622us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.850727 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.856504 13861 tablet_bootstrap.cc:654] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.856141 13867 raft_consensus.cc:504] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.850273 13729 rpcz_store.cc:276] 0503 02:06:59.763097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.577891 (+814794us) service_pool.cc:225] Handling call
0503 02:07:00.648874 (+ 70983us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.648978 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.799266 (+150288us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.849426 (+ 50160us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.849838 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.829080 13727 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 38) took 1056 ms. Trace:
I20240503 02:07:00.858225 13727 rpcz_store.cc:276] 0503 02:06:59.759896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.513750 (+753854us) service_pool.cc:225] Handling call
0503 02:07:00.526523 (+ 12773us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.526614 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.740003 (+213389us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.815821 (+ 75818us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.816224 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.862304 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6b185135347d58dfb8ddd0b758e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.861377 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97598aefb41c4115a37385063f8d0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.872267 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74128b771f146e78210a5b8dcf823a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.873396 13650 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 40) took 1042 ms. Trace:
I20240503 02:07:00.873836 13650 rpcz_store.cc:276] 0503 02:06:59.830520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.724247 (+893727us) service_pool.cc:225] Handling call
0503 02:07:00.726072 (+ 1825us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.726160 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.840054 (+113894us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.857598 (+ 17544us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.873315 (+ 15717us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.875854 13650 tablet_service.cc:1467] Processing CreateTablet for tablet f0d6b185135347d58dfb8ddd0b758e17 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 02:07:00.877060 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b664b6227e0f4268a89935f722d56c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.878078 13806 tablet_service.cc:1467] Processing CreateTablet for tablet b2365517fa2742fead22e9bceb13af2d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 02:07:00.879467 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2365517fa2742fead22e9bceb13af2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.885802 13807 tablet_service.cc:1467] Processing CreateTablet for tablet 7e254fe56b844e41b2abbf3626923726 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 02:07:00.887022 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e254fe56b844e41b2abbf3626923726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.890004 13652 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 02:07:00.891647 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 1d086d958d664a1ca6d0a0cc9324f480 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240503 02:07:00.859073 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 02:07:00.895902 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.897785 13799 tablet_service.cc:1467] Processing CreateTablet for tablet feefafcc54784de88a55414462073851 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 02:07:00.898994 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feefafcc54784de88a55414462073851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.908604 13655 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 35) took 1079 ms. Trace:
I20240503 02:07:00.909050 13655 rpcz_store.cc:276] 0503 02:06:59.828689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.659829 (+831140us) service_pool.cc:225] Handling call
0503 02:07:00.710657 (+ 50828us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.713895 (+ 3238us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.824143 (+110248us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.886150 (+ 62007us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.908542 (+ 22392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.912006 13651 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 39) took 1081 ms. Trace:
I20240503 02:07:00.913322 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.915594 13860 tablet_bootstrap.cc:654] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:00.912389 13651 rpcz_store.cc:276] 0503 02:06:59.830176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.690646 (+860470us) service_pool.cc:225] Handling call
0503 02:07:00.692402 (+ 1756us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.692506 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.805659 (+113153us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.894564 (+ 88905us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.911939 (+ 17375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.860132 13727 tablet_service.cc:1467] Processing CreateTablet for tablet a321281efc6f4adf8d7fc972c85eef37 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 02:07:00.918238 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a321281efc6f4adf8d7fc972c85eef37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.918840 13651 tablet_service.cc:1467] Processing CreateTablet for tablet d67aeb523f5b4da696a2135fb1acb234 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 02:07:00.917142 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0d6b185135347d58dfb8ddd0b758e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.923367 13803 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac94ddbd01d4794b3711a772388f6d9 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 02:07:00.924661 13803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac94ddbd01d4794b3711a772388f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.932549 13861 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:00.933084 13861 ts_tablet_manager.cc:1367] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.003s
I20240503 02:07:00.933753 13728 tablet_service.cc:1467] Processing CreateTablet for tablet b2365517fa2742fead22e9bceb13af2d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 02:07:00.934978 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2365517fa2742fead22e9bceb13af2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.936982 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d086d958d664a1ca6d0a0cc9324f480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.936326 13655 tablet_service.cc:1467] Processing CreateTablet for tablet b664b6227e0f4268a89935f722d56c8f (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240503 02:07:00.942502 13729 tablet_service.cc:1467] Processing CreateTablet for tablet a2243e6f9b764eda92cfa44cb769fde6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 02:07:00.943707 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2243e6f9b764eda92cfa44cb769fde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.938205 13656 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 38) took 1107 ms. Trace:
I20240503 02:07:00.944308 13656 rpcz_store.cc:276] 0503 02:06:59.829770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.688804 (+859034us) service_pool.cc:225] Handling call
0503 02:07:00.764924 (+ 76120us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.765019 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.889664 (+124645us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.912580 (+ 22916us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.937051 (+ 24471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.946135 13726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 40) took 1179 ms. Trace:
I20240503 02:07:00.948048 13726 rpcz_store.cc:276] 0503 02:06:59.766310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.617935 (+851625us) service_pool.cc:225] Handling call
0503 02:07:00.632790 (+ 14855us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.632885 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.826974 (+194089us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.937128 (+110154us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.946078 (+ 8950us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.846988 13730 tablet_service.cc:1467] Processing CreateTablet for tablet d67aeb523f5b4da696a2135fb1acb234 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 02:07:00.950592 13659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 36) took 1121 ms. Trace:
I20240503 02:07:00.954814 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24625c57a084e26a113238421e92823" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:00.955380 13743 raft_consensus.cc:3037] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:00.949856 13656 tablet_service.cc:1467] Processing CreateTablet for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 02:07:00.947474 13654 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 42) took 1116 ms. Trace:
I20240503 02:07:00.957242 13654 rpcz_store.cc:276] 0503 02:06:59.831133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.797393 (+966260us) service_pool.cc:225] Handling call
0503 02:07:00.800316 (+ 2923us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.800489 (+ 173us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.902423 (+101934us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.946978 (+ 44555us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.947418 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.951032 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67aeb523f5b4da696a2135fb1acb234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.947389 13804 tablet_service.cc:1467] Processing CreateTablet for tablet af476a0cb51d478b952e5c4fb44447e2 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 02:07:00.950973 13726 tablet_service.cc:1467] Processing CreateTablet for tablet 7e254fe56b844e41b2abbf3626923726 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 02:07:00.952427 13653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 34) took 1124 ms. Trace:
I20240503 02:07:00.953519 13858 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:00.953547 13867 leader_election.cc:288] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:00.945873 13800 tablet_service.cc:1467] Processing CreateTablet for tablet 96e9257172234c59a0ad9b65af11d512 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 02:07:00.954535 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24625c57a084e26a113238421e92823" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:00.935674 13861 raft_consensus.cc:348] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.955097 13659 rpcz_store.cc:276] 0503 02:06:59.829062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.646400 (+817338us) service_pool.cc:225] Handling call
0503 02:07:00.769708 (+123308us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.769793 (+ 85us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.847558 (+ 77765us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.931362 (+ 83804us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.950521 (+ 19159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.964134 13861 raft_consensus.cc:374] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:00.964486 13861 raft_consensus.cc:729] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
W20240503 02:07:00.964978 13783 leader_election.cc:341] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:00.965174 13861 consensus_queue.cc:260] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:00.966691 13654 tablet_service.cc:1467] Processing CreateTablet for tablet a321281efc6f4adf8d7fc972c85eef37 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 02:07:00.979454 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e254fe56b844e41b2abbf3626923726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.980537 13731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 44) took 1171 ms. Trace:
I20240503 02:07:00.984961 13731 rpcz_store.cc:276] 0503 02:06:59.809097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.653240 (+844143us) service_pool.cc:225] Handling call
0503 02:07:00.654915 (+ 1675us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.655004 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.793422 (+138418us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.939428 (+146006us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.980230 (+ 40802us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b58002d0098. stack: 00007f701e23cba3 00007f701e23caa0 00007f701dba1584 00007f70278b79e4 00007f70278b7979 00007f701e0b5b51 00007f701e2373b1 00007f701e2379e2 00007f70271525f9 00007f7026066268 00007f7026129d33 00007f7026125b4f 00007f7026126301 00007f70260d7619 00007f701f6d5c64 00007f701f6d5bf2
0503 02:07:00.980480 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11210,"spinlock_wait_cycles":34857216}
I20240503 02:07:00.987228 13731 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac94ddbd01d4794b3711a772388f6d9 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 02:07:00.988546 13731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac94ddbd01d4794b3711a772388f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.962121 13653 rpcz_store.cc:276] 0503 02:06:59.828395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.655730 (+827335us) service_pool.cc:225] Handling call
0503 02:07:00.657434 (+ 1704us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.657526 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.784124 (+126598us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.951982 (+167858us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.952373 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.994778 13653 tablet_service.cc:1467] Processing CreateTablet for tablet a2243e6f9b764eda92cfa44cb769fde6 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 02:07:00.994757 13650 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 46) took 1003 ms. Trace:
I20240503 02:07:00.996670 13650 rpcz_store.cc:276] 0503 02:06:59.991221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.874838 (+883617us) service_pool.cc:225] Handling call
0503 02:07:00.876734 (+ 1896us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.907485 (+ 30751us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.961584 (+ 54099us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.989885 (+ 28301us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:00.994625 (+ 4740us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:00.997984 13729 tablet_service.cc:1467] Processing CreateTablet for tablet fff18016e2e947a987c7403a319f13bc (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240503 02:07:00.999243 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff18016e2e947a987c7403a319f13bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.998438 13650 tablet_service.cc:1467] Processing CreateTablet for tablet b2365517fa2742fead22e9bceb13af2d (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 02:07:00.974954 13858 ts_tablet_manager.cc:1367] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.422s user 0.038s sys 0.057s
I20240503 02:07:01.007483 13858 raft_consensus.cc:348] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.008293 13858 raft_consensus.cc:374] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.009907 13858 raft_consensus.cc:729] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.012845 13743 raft_consensus.cc:2454] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.033867 13781 leader_election.cc:302] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.034978 13867 raft_consensus.cc:2781] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.035463 13867 raft_consensus.cc:686] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.036370 13867 consensus_queue.cc:237] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.009267 13801 tablet_service.cc:1467] Processing CreateTablet for tablet fff18016e2e947a987c7403a319f13bc (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240503 02:07:01.020661 13659 tablet_service.cc:1467] Processing CreateTablet for tablet 7e254fe56b844e41b2abbf3626923726 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 02:07:01.025240 13725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 36) took 1271 ms. Trace:
I20240503 02:07:01.025871 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af476a0cb51d478b952e5c4fb44447e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:00.976240 13651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67aeb523f5b4da696a2135fb1acb234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.048799 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b664b6227e0f4268a89935f722d56c8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:07:01.050112 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:01.049870 13858 consensus_queue.cc:260] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.052978 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a321281efc6f4adf8d7fc972c85eef37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.053025 13802 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42260 (request call id 42) took 1045 ms. Trace:
I20240503 02:07:01.053481 13802 rpcz_store.cc:276] 0503 02:07:00.007588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.628414 (+620826us) service_pool.cc:225] Handling call
0503 02:07:00.670459 (+ 42045us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.670563 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.931116 (+260553us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.032801 (+101685us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.052968 (+ 20167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.053447 13801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff18016e2e947a987c7403a319f13bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.058375 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2243e6f9b764eda92cfa44cb769fde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.060371 13866 raft_consensus.cc:482] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.060940 13861 ts_tablet_manager.cc:1398] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.128s user 0.004s sys 0.003s
I20240503 02:07:01.060856 13866 raft_consensus.cc:504] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.062211 13734 tablet_service.cc:1467] Processing CreateTablet for tablet feefafcc54784de88a55414462073851 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 02:07:01.063015 13866 leader_election.cc:288] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.063424 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feefafcc54784de88a55414462073851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.061064 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c6c7248bdc40acad0bd1dd3ad6cb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.069783 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.070854 13727 tablet_service.cc:1467] Processing CreateTablet for tablet af476a0cb51d478b952e5c4fb44447e2 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 02:07:01.071069 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:01.071424 13635 leader_election.cc:341] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.072036 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af476a0cb51d478b952e5c4fb44447e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.041536 13725 rpcz_store.cc:276] 0503 02:06:59.754111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.453723 (+699612us) service_pool.cc:225] Handling call
0503 02:07:00.511590 (+ 57867us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.511704 (+ 114us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.840559 (+328855us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:00.961711 (+121152us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.025156 (+ 63445us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:07:01.072643 13632 leader_election.cc:341] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.073225 13632 leader_election.cc:302] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.075019 13866 raft_consensus.cc:2726] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.076965 13861 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.076174 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e9257172234c59a0ad9b65af11d512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.092238 13858 ts_tablet_manager.cc:1398] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.087s user 0.004s sys 0.002s
I20240503 02:07:01.093618 13566 catalog_manager.cc:5484] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.096017 13729 tablet_service.cc:1467] Processing CreateTablet for tablet 96e9257172234c59a0ad9b65af11d512 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 02:07:01.097373 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e9257172234c59a0ad9b65af11d512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.104053 13860 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.104605 13860 ts_tablet_manager.cc:1367] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.424s user 0.019s sys 0.023s
I20240503 02:07:01.107350 13860 raft_consensus.cc:348] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.108161 13858 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.111503 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2365517fa2742fead22e9bceb13af2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.121583 13658 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 41) took 1290 ms. Trace:
I20240503 02:07:01.122004 13658 rpcz_store.cc:276] 0503 02:06:59.830825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.793622 (+962797us) service_pool.cc:225] Handling call
0503 02:07:00.795334 (+ 1712us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.795437 (+ 103us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.004080 (+208643us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.062269 (+ 58189us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.121243 (+ 58974us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b5800004598. stack: 00007f701e23cba3 00007f701e23caa0 00007f701dba1584 00007f70278b79e4 00007f70278b7979 00007f701e0b5b51 00007f701e2373b1 00007f701e2379e2 00007f70271525f9 00007f7026066268 00007f7026129d33 00007f7026125b4f 00007f7026126301 00007f70260d7619 00007f701f6d5c64 00007f701f6d5bf2
0503 02:07:01.121521 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9037568}
I20240503 02:07:01.122911 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e254fe56b844e41b2abbf3626923726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.124140 13658 tablet_service.cc:1467] Processing CreateTablet for tablet feefafcc54784de88a55414462073851 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 02:07:01.125428 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feefafcc54784de88a55414462073851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.135324 13657 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 43) took 1303 ms. Trace:
I20240503 02:07:01.135771 13657 rpcz_store.cc:276] 0503 02:06:59.831467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.861836 (+1030369us) service_pool.cc:225] Handling call
0503 02:07:00.919879 (+ 58043us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.919985 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.074566 (+154581us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.134779 (+ 60213us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.135251 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.159373 13866 raft_consensus.cc:482] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.159894 13866 raft_consensus.cc:504] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.160439 13733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 47) took 1124 ms. Trace:
I20240503 02:07:01.112181 13860 raft_consensus.cc:374] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.172600 13734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 54) took 1015 ms. Trace:
I20240503 02:07:01.173566 13866 leader_election.cc:288] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.174732 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.175158 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.182893 13726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 53) took 1028 ms. Trace:
I20240503 02:07:01.183267 13726 rpcz_store.cc:276] 0503 02:07:00.154143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.949516 (+795373us) service_pool.cc:225] Handling call
0503 02:07:00.979002 (+ 29486us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.979098 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.077209 (+ 98111us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.182447 (+105238us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.182837 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240503 02:07:01.184123 13632 leader_election.cc:341] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:01.185420 13635 leader_election.cc:341] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.185855 13635 leader_election.cc:302] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.186599 13730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 48) took 1133 ms. Trace:
I20240503 02:07:01.186972 13730 rpcz_store.cc:276] 0503 02:07:00.053601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.845169 (+791568us) service_pool.cc:225] Handling call
0503 02:07:00.950606 (+105437us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.950690 (+ 84us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.075167 (+124477us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.186137 (+110970us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.186543 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.164414 13733 rpcz_store.cc:276] 0503 02:07:00.036064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.832959 (+796895us) service_pool.cc:225] Handling call
0503 02:07:00.834749 (+ 1790us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.834841 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:00.969013 (+134172us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.159906 (+190893us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.160374 (+ 468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.167119 13860 raft_consensus.cc:729] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.181394 13652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 44) took 1349 ms. Trace:
I20240503 02:07:01.183735 13858 tablet_bootstrap.cc:654] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.188647 13866 raft_consensus.cc:2726] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.188640 13860 consensus_queue.cc:260] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.191267 13652 rpcz_store.cc:276] 0503 02:06:59.831813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.889101 (+1057288us) service_pool.cc:225] Handling call
0503 02:07:00.912812 (+ 23711us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.912916 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.050653 (+137737us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.145895 (+ 95242us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.181329 (+ 35434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.192315 13860 ts_tablet_manager.cc:1398] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.087s user 0.005s sys 0.002s
I20240503 02:07:01.193506 13860 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.193843 13652 tablet_service.cc:1467] Processing CreateTablet for tablet fff18016e2e947a987c7403a319f13bc (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240503 02:07:01.176692 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac94ddbd01d4794b3711a772388f6d9 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 02:07:01.195065 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff18016e2e947a987c7403a319f13bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.200006 13860 tablet_bootstrap.cc:654] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.200917 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac94ddbd01d4794b3711a772388f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.205408 13861 tablet_bootstrap.cc:654] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.201169 13734 rpcz_store.cc:276] 0503 02:07:00.157488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.973788 (+816300us) service_pool.cc:225] Handling call
0503 02:07:01.062986 (+ 89198us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.063077 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.135794 (+ 72717us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.172017 (+ 36223us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.172526 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.176576 13732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 49) took 1114 ms. Trace:
I20240503 02:07:01.211869 13732 rpcz_store.cc:276] 0503 02:07:00.062117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.858149 (+796032us) service_pool.cc:225] Handling call
0503 02:07:00.895465 (+ 37316us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.895557 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.083753 (+188196us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.176101 (+ 92348us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.176515 (+ 414us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.224581 13728 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54470 (request call id 52) took 1077 ms. Trace:
I20240503 02:07:01.225064 13728 rpcz_store.cc:276] 0503 02:07:00.147064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.932862 (+785798us) service_pool.cc:225] Handling call
0503 02:07:00.934544 (+ 1682us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.934636 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.116537 (+181901us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.224087 (+107550us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.224512 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.231158 13858 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.231628 13858 ts_tablet_manager.cc:1367] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.009s
I20240503 02:07:01.232388 13658 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 54) took 1075 ms. Trace:
I20240503 02:07:01.232815 13658 rpcz_store.cc:276] 0503 02:07:00.156883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:01.123146 (+966263us) service_pool.cc:225] Handling call
0503 02:07:01.125006 (+ 1860us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.125094 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.201283 (+ 76189us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.231869 (+ 30586us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.232323 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.234246 13656 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 49) took 1178 ms. Trace:
I20240503 02:07:01.234623 13656 rpcz_store.cc:276] 0503 02:07:00.055949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.949036 (+893087us) service_pool.cc:225] Handling call
0503 02:07:00.965855 (+ 16819us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b5800004598. stack: 00007f701e23cba3 00007f701e23caa0 00007f701dba1584 00007f70278b79e4 00007f70278b7979 00007f701e0b5b51 00007f701e2373b1 00007f701e237262 00007f7027129889 00007f70260663c0 00007f7026148791 00007f7026125f03 00007f70260d7619 00007f701f6d5c64 00007f701f6d5bf2 00007f701f6d5b21
0503 02:07:00.990534 (+ 24679us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.990629 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.185014 (+194385us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.233804 (+ 48790us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.234190 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8900992}
I20240503 02:07:01.236197 13658 tablet_service.cc:1467] Processing CreateTablet for tablet 96e9257172234c59a0ad9b65af11d512 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 02:07:01.236909 13654 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 50) took 1166 ms. Trace:
I20240503 02:07:01.237285 13655 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 47) took 1238 ms. Trace:
I20240503 02:07:01.238229 13653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 51) took 1153 ms. Trace:
I20240503 02:07:01.236457 13656 tablet_service.cc:1467] Processing CreateTablet for tablet af476a0cb51d478b952e5c4fb44447e2 (DEFAULT_TABLE table=test-workload [id=817018b9773e40f5b40c601196a15177]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 02:07:01.237692 13654 rpcz_store.cc:276] 0503 02:07:00.070215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.961735 (+891520us) service_pool.cc:225] Handling call
0503 02:07:00.990408 (+ 28673us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800004598. stack: 00007f701e23cba3 00007f701e23caa0 00007f701dba1584 00007f70278b79e4 00007f70278b7979 00007f701e0b5b51 00007f701e2376dc 00007f701e23729b 00007f7027129889 00007f70260663c0 00007f7026148791 00007f7026125f03 00007f70260d7619 00007f701f6d5c64 00007f701f6d5bf2 00007f701f6d5b21
0503 02:07:00.993542 (+ 3134us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.993723 (+ 181us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.147993 (+154270us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.215046 (+ 67053us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.236843 (+ 21797us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31807744}
I20240503 02:07:01.240222 13656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af476a0cb51d478b952e5c4fb44447e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.237704 13655 rpcz_store.cc:276] 0503 02:06:59.998857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.910657 (+911800us) service_pool.cc:225] Handling call
0503 02:07:00.969672 (+ 59015us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.969798 (+ 126us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.175279 (+205481us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.229024 (+ 53745us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.237228 (+ 8204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1199360}
I20240503 02:07:01.227375 13651 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 48) took 1187 ms. Trace:
I20240503 02:07:01.242444 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96e9257172234c59a0ad9b65af11d512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:01.241916 13651 rpcz_store.cc:276] 0503 02:07:00.039546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.917999 (+878453us) service_pool.cc:225] Handling call
0503 02:07:00.919625 (+ 1626us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.919707 (+ 82us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.109104 (+189397us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.218310 (+109206us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.227036 (+ 8726us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.238600 13653 rpcz_store.cc:276] 0503 02:07:00.084846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.993809 (+908963us) service_pool.cc:225] Handling call
0503 02:07:00.996090 (+ 2281us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.996179 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.127156 (+130977us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.237768 (+110612us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.238175 (+ 407us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.228780 13807 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42260 (request call id 53) took 1072 ms. Trace:
I20240503 02:07:01.234220 13858 raft_consensus.cc:348] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.244437 13807 rpcz_store.cc:276] 0503 02:07:00.155491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.884945 (+729454us) service_pool.cc:225] Handling call
0503 02:07:00.886609 (+ 1664us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:00.886697 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.099869 (+213172us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.218995 (+119126us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.228132 (+ 9137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.245594 13659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 53) took 1092 ms. Trace:
I20240503 02:07:01.244817 13858 raft_consensus.cc:374] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.249842 13659 rpcz_store.cc:276] 0503 02:07:00.152805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:01.019748 (+866943us) service_pool.cc:225] Handling call
0503 02:07:01.067594 (+ 47846us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.073433 (+ 5839us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.189878 (+116445us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.245080 (+ 55202us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.245530 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.254472 13861 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.252960 13650 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 52) took 1140 ms. Trace:
I20240503 02:07:01.255007 13650 rpcz_store.cc:276] 0503 02:07:00.112907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:00.997621 (+884714us) service_pool.cc:225] Handling call
0503 02:07:01.002458 (+ 4837us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.002548 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.226851 (+224303us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.252531 (+ 25680us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.252895 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.255015 13861 ts_tablet_manager.cc:1367] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.178s user 0.012s sys 0.000s
I20240503 02:07:01.256515 13860 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.257112 13860 ts_tablet_manager.cc:1367] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.010s
I20240503 02:07:01.256506 13657 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 55) took 1004 ms. Trace:
I20240503 02:07:01.259591 13860 raft_consensus.cc:348] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.260327 13860 raft_consensus.cc:374] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.260640 13860 raft_consensus.cc:729] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.261533 13860 consensus_queue.cc:260] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.250061 13858 raft_consensus.cc:729] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.259837 13657 rpcz_store.cc:276] 0503 02:07:00.252361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:01.175705 (+923344us) service_pool.cc:225] Handling call
0503 02:07:01.200335 (+ 24630us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.200492 (+ 157us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.225723 (+ 25231us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.256048 (+ 30325us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.256448 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.262406 13652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54568 (request call id 56) took 1006 ms. Trace:
I20240503 02:07:01.263371 13652 rpcz_store.cc:276] 0503 02:07:00.256052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:07:01.192463 (+936411us) service_pool.cc:225] Handling call
0503 02:07:01.194618 (+ 2155us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0503 02:07:01.194710 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0503 02:07:01.225274 (+ 30564us) tablet_metadata.cc:641] Metadata flushed
0503 02:07:01.261903 (+ 36629us) tablet_replica.cc:205] Creating consensus instance
0503 02:07:01.262342 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240503 02:07:01.265192 13858 consensus_queue.cc:260] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.266682 13861 raft_consensus.cc:348] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.267367 13861 raft_consensus.cc:374] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.267494 13858 ts_tablet_manager.cc:1398] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20240503 02:07:01.267881 13860 ts_tablet_manager.cc:1398] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240503 02:07:01.268853 13860 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.274325 13860 tablet_bootstrap.cc:654] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.275190 13865 raft_consensus.cc:482] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.275652 13865 raft_consensus.cc:504] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.276906 13861 raft_consensus.cc:729] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.278038 13867 raft_consensus.cc:482] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.278457 13867 raft_consensus.cc:504] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.279079 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:01.280294 13710 leader_election.cc:341] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.280373 13867 leader_election.cc:288] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.277670 13865 leader_election.cc:288] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.277712 13861 consensus_queue.cc:260] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.281469 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:01.284790 13709 leader_election.cc:341] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.284888 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.285234 13709 leader_election.cc:302] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.285550 13668 raft_consensus.cc:2454] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.286367 13865 raft_consensus.cc:2726] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.281592 13858 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.283000 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.286831 13783 leader_election.cc:302] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.287730 13867 raft_consensus.cc:2781] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.288152 13860 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
W20240503 02:07:01.288045 13781 leader_election.cc:341] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.288105 13867 raft_consensus.cc:482] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.291494 13867 raft_consensus.cc:3037] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.291587 13860 ts_tablet_manager.cc:1367] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.004s
I20240503 02:07:01.285249 13861 ts_tablet_manager.cc:1398] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240503 02:07:01.294945 13860 raft_consensus.cc:348] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.295764 13860 raft_consensus.cc:374] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.296103 13860 raft_consensus.cc:729] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.296660 13861 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.297086 13860 consensus_queue.cc:260] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.298573 13867 raft_consensus.cc:504] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.299788 13860 ts_tablet_manager.cc:1398] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 02:07:01.300767 13860 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.301304 13867 leader_election.cc:288] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.301564 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.302078 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.302632 13668 raft_consensus.cc:3037] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:01.303969 13781 leader_election.cc:341] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.306031 13860 tablet_bootstrap.cc:654] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.309689 13668 raft_consensus.cc:2454] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.310860 13783 leader_election.cc:302] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:01.311322 13861 tablet_bootstrap.cc:654] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.312716 13867 raft_consensus.cc:2781] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.314046 13867 raft_consensus.cc:686] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.314975 13867 consensus_queue.cc:237] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.316073 13858 tablet_bootstrap.cc:654] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.317688 13860 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.318220 13860 ts_tablet_manager.cc:1367] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240503 02:07:01.321166 13860 raft_consensus.cc:348] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.322098 13860 raft_consensus.cc:374] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.322472 13860 raft_consensus.cc:729] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.323241 13860 consensus_queue.cc:260] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.325690 13860 ts_tablet_manager.cc:1398] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:07:01.326750 13860 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.329241 13566 catalog_manager.cc:5484] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.333197 13860 tablet_bootstrap.cc:654] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.339634 13861 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.339908 13858 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.340453 13858 ts_tablet_manager.cc:1367] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.010s
I20240503 02:07:01.343300 13858 raft_consensus.cc:348] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.344079 13858 raft_consensus.cc:374] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.344383 13858 raft_consensus.cc:729] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.345208 13858 consensus_queue.cc:260] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.347101 13858 ts_tablet_manager.cc:1398] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:01.347544 13860 tablet_bootstrap.cc:492] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.347947 13860 ts_tablet_manager.cc:1367] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20240503 02:07:01.349954 13858 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.351408 13860 raft_consensus.cc:348] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.352859 13861 ts_tablet_manager.cc:1367] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.009s
I20240503 02:07:01.351953 13860 raft_consensus.cc:374] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.353344 13860 raft_consensus.cc:729] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.354115 13860 consensus_queue.cc:260] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.356069 13858 tablet_bootstrap.cc:654] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.355674 13861 raft_consensus.cc:348] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.356456 13861 raft_consensus.cc:374] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.356767 13861 raft_consensus.cc:729] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.357508 13861 consensus_queue.cc:260] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.359037 13860 ts_tablet_manager.cc:1398] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240503 02:07:01.359774 13861 ts_tablet_manager.cc:1398] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240503 02:07:01.360004 13860 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.364825 13867 raft_consensus.cc:482] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.365317 13867 raft_consensus.cc:504] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.367012 13861 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.367244 13867 leader_election.cc:288] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.367964 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a20af2756040cea6c15f6ace3912bb" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.369484 13860 tablet_bootstrap.cc:654] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.369477 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a20af2756040cea6c15f6ace3912bb" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.370224 13743 raft_consensus.cc:2454] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.371469 13781 leader_election.cc:302] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.372212 13867 raft_consensus.cc:2781] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.372551 13867 raft_consensus.cc:482] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.372895 13867 raft_consensus.cc:3037] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:01.372951 13783 leader_election.cc:341] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.373548 13861 tablet_bootstrap.cc:654] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.379602 13867 raft_consensus.cc:504] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.382339 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a20af2756040cea6c15f6ace3912bb" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.382606 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a20af2756040cea6c15f6ace3912bb" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.383124 13743 raft_consensus.cc:3037] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:01.383834 13783 leader_election.cc:341] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.388904 13867 leader_election.cc:288] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.388999 13743 raft_consensus.cc:2454] T c4a20af2756040cea6c15f6ace3912bb P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.390090 13858 tablet_bootstrap.cc:492] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.390112 13781 leader_election.cc:302] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.390812 13858 ts_tablet_manager.cc:1367] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.041s user 0.021s sys 0.019s
I20240503 02:07:01.391466 13867 raft_consensus.cc:2781] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.391963 13867 raft_consensus.cc:686] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.392796 13867 consensus_queue.cc:237] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.393826 13858 raft_consensus.cc:348] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.394268 13860 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.394580 13858 raft_consensus.cc:374] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.394805 13860 ts_tablet_manager.cc:1367] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.009s
I20240503 02:07:01.394876 13858 raft_consensus.cc:729] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.395643 13858 consensus_queue.cc:260] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.397367 13860 raft_consensus.cc:348] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.397835 13858 ts_tablet_manager.cc:1398] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:07:01.398056 13860 raft_consensus.cc:374] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.398334 13860 raft_consensus.cc:729] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.398952 13858 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.398944 13860 consensus_queue.cc:260] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.402441 13861 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.402925 13861 ts_tablet_manager.cc:1367] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.002s
I20240503 02:07:01.405076 13858 tablet_bootstrap.cc:654] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.405522 13861 raft_consensus.cc:348] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.406270 13861 raft_consensus.cc:374] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.406581 13861 raft_consensus.cc:729] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.407275 13861 consensus_queue.cc:260] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.409687 13860 ts_tablet_manager.cc:1398] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240503 02:07:01.410786 13860 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.416426 13858 tablet_bootstrap.cc:492] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.417047 13858 ts_tablet_manager.cc:1367] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240503 02:07:01.417579 13883 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:01.419546 13860 tablet_bootstrap.cc:654] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.419591 13858 raft_consensus.cc:348] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.420387 13858 raft_consensus.cc:374] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.420670 13858 raft_consensus.cc:729] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.421375 13858 consensus_queue.cc:260] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.422974 13861 ts_tablet_manager.cc:1398] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20240503 02:07:01.423527 13858 ts_tablet_manager.cc:1398] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:01.424620 13858 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.425056 13861 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.431005 13858 tablet_bootstrap.cc:654] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.432260 13668 raft_consensus.cc:3037] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.432917 13861 tablet_bootstrap.cc:654] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.437461 13565 catalog_manager.cc:5484] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.441412 13883 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:01.445326 13860 tablet_bootstrap.cc:492] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.445755 13860 ts_tablet_manager.cc:1367] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.000s
I20240503 02:07:01.443382 13861 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.446563 13861 ts_tablet_manager.cc:1367] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20240503 02:07:01.448575 13867 raft_consensus.cc:482] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.449041 13867 raft_consensus.cc:504] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.444245 13858 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.450981 13867 leader_election.cc:288] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.451092 13858 ts_tablet_manager.cc:1367] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.012s
I20240503 02:07:01.452067 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.452710 13743 raft_consensus.cc:2454] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.453645 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.454241 13669 raft_consensus.cc:2454] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.454202 13858 raft_consensus.cc:348] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.455479 13858 raft_consensus.cc:374] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.455391 13781 leader_election.cc:302] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.455809 13858 raft_consensus.cc:729] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.456374 13861 raft_consensus.cc:348] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.456447 13858 consensus_queue.cc:260] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.457247 13861 raft_consensus.cc:374] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.457645 13861 raft_consensus.cc:729] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.458369 13861 consensus_queue.cc:260] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.460399 13861 ts_tablet_manager.cc:1398] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 02:07:01.462023 13858 ts_tablet_manager.cc:1398] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240503 02:07:01.462986 13858 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.463271 13860 raft_consensus.cc:348] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.463949 13860 raft_consensus.cc:374] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.464208 13860 raft_consensus.cc:729] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.466634 13860 consensus_queue.cc:260] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.468071 13858 tablet_bootstrap.cc:654] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.472589 13861 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.474848 13867 raft_consensus.cc:2781] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.475215 13867 raft_consensus.cc:482] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.475548 13867 raft_consensus.cc:3037] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.479559 13861 tablet_bootstrap.cc:654] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.483139 13865 raft_consensus.cc:482] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.483589 13865 raft_consensus.cc:504] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.485491 13865 leader_election.cc:288] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.485390 13867 raft_consensus.cc:504] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.487144 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.487798 13818 raft_consensus.cc:2454] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:01.487975 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.488523 13669 raft_consensus.cc:3037] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.489815 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.491591 13710 leader_election.cc:302] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
W20240503 02:07:01.492874 13709 leader_election.cc:341] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.497581 13910 raft_consensus.cc:482] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.498682 13865 raft_consensus.cc:2781] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.499087 13865 raft_consensus.cc:482] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.499428 13865 raft_consensus.cc:3037] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.504297 13860 ts_tablet_manager.cc:1398] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.058s user 0.006s sys 0.002s
I20240503 02:07:01.505242 13860 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.508291 13858 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.508831 13858 ts_tablet_manager.cc:1367] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.046s user 0.028s sys 0.010s
I20240503 02:07:01.509792 13916 raft_consensus.cc:482] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.510433 13867 leader_election.cc:288] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.510896 13860 tablet_bootstrap.cc:654] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.511536 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ec7430d7f43baadcc6dd1296d7b28" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.511386 13858 raft_consensus.cc:348] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.512794 13858 raft_consensus.cc:374] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.513229 13858 raft_consensus.cc:729] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.513931 13858 consensus_queue.cc:260] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.516263 13744 raft_consensus.cc:3037] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.512334 13916 raft_consensus.cc:504] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.498653 13883 raft_consensus.cc:482] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.525938 13865 raft_consensus.cc:504] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.498108 13910 raft_consensus.cc:504] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.527627 13883 raft_consensus.cc:504] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.523341 13669 raft_consensus.cc:2454] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.532109 13783 leader_election.cc:302] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.532428 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.532905 13883 leader_election.cc:288] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.533723 13867 raft_consensus.cc:2781] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.534487 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.534929 13867 raft_consensus.cc:686] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.535084 13858 ts_tablet_manager.cc:1398] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.026s user 0.005s sys 0.010s
I20240503 02:07:01.535234 13742 raft_consensus.cc:2454] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.536199 13916 leader_election.cc:288] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.535857 13867 consensus_queue.cc:237] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.537665 13858 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.538012 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.539448 13781 leader_election.cc:302] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.540408 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.541023 13667 raft_consensus.cc:2454] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
W20240503 02:07:01.542078 13709 leader_election.cc:341] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.533506 13744 raft_consensus.cc:2454] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.542649 13858 tablet_bootstrap.cc:654] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.543285 13910 leader_election.cc:288] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.543843 13783 leader_election.cc:302] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.544255 13910 raft_consensus.cc:2781] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.544581 13910 raft_consensus.cc:482] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.546809 13883 raft_consensus.cc:2781] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.548648 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9c8e98bdb04950a65590a69bf0ac19" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.549048 13883 raft_consensus.cc:482] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.549387 13883 raft_consensus.cc:3037] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.549502 13667 raft_consensus.cc:2379] T fe9c8e98bdb04950a65590a69bf0ac19 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:01.548651 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.551183 13709 leader_election.cc:302] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.551429 13669 raft_consensus.cc:2454] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.552122 13918 raft_consensus.cc:3037] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.550400 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.544911 13910 raft_consensus.cc:3037] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.556285 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:01.556967 13818 raft_consensus.cc:3037] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.572228 13818 raft_consensus.cc:2454] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:01.573485 13861 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.573683 13710 leader_election.cc:302] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.573978 13861 ts_tablet_manager.cc:1367] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.003s
I20240503 02:07:01.574770 13921 raft_consensus.cc:2781] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.575347 13921 raft_consensus.cc:686] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:01.576390 13861 raft_consensus.cc:348] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.577750 13861 raft_consensus.cc:374] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.576283 13921 consensus_queue.cc:237] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.578047 13861 raft_consensus.cc:729] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.578701 13861 consensus_queue.cc:260] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.559808 13865 leader_election.cc:288] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.580828 13861 ts_tablet_manager.cc:1398] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:07:01.582085 13883 raft_consensus.cc:504] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.582612 13861 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.587267 13910 raft_consensus.cc:504] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.588192 13861 tablet_bootstrap.cc:654] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.589951 13918 raft_consensus.cc:2726] T fe9c8e98bdb04950a65590a69bf0ac19 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.591475 13910 leader_election.cc:288] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.592240 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.592900 13669 raft_consensus.cc:3037] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.601033 13669 raft_consensus.cc:2454] T c4f6dafbb5974de589d4064a5f979f01 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.602392 13783 leader_election.cc:302] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.608712 13883 leader_election.cc:288] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.609393 13860 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.609838 13860 ts_tablet_manager.cc:1367] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.105s user 0.013s sys 0.001s
I20240503 02:07:01.606997 13910 raft_consensus.cc:2781] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:07:01.607115 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 55471.2us
I20240503 02:07:01.611133 13567 catalog_manager.cc:5484] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:01.613370 13910 raft_consensus.cc:686] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.614149 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f6dafbb5974de589d4064a5f979f01" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.614734 13744 raft_consensus.cc:3037] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.614293 13910 consensus_queue.cc:237] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.617115 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.618726 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.619290 13669 raft_consensus.cc:3037] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.619998 13860 raft_consensus.cc:348] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.620756 13860 raft_consensus.cc:374] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.621032 13860 raft_consensus.cc:729] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.621649 13860 consensus_queue.cc:260] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.622732 13921 raft_consensus.cc:482] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.623216 13921 raft_consensus.cc:504] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.625277 13921 leader_election.cc:288] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.626137 13858 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.626561 13858 ts_tablet_manager.cc:1367] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.089s user 0.024s sys 0.016s
I20240503 02:07:01.626670 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.626868 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:01.628286 13710 leader_election.cc:341] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:01.628911 13709 leader_election.cc:341] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.628893 13858 raft_consensus.cc:348] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.629714 13858 raft_consensus.cc:374] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.630008 13858 raft_consensus.cc:729] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.629532 13709 leader_election.cc:302] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.630614 13858 consensus_queue.cc:260] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:01.628846 13781 leader_election.cc:341] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.631562 13918 raft_consensus.cc:2726] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:07:01.632243 13781 leader_election.cc:341] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.632571 13858 ts_tablet_manager.cc:1398] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:07:01.633289 13860 ts_tablet_manager.cc:1398] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240503 02:07:01.633395 13858 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.634276 13860 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.643694 13744 raft_consensus.cc:2454] T c4f6dafbb5974de589d4064a5f979f01 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.647514 13669 raft_consensus.cc:2454] T c782825ce834414baad8c3f6b6642e5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.648070 13913 raft_consensus.cc:482] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.648878 13858 tablet_bootstrap.cc:654] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.650688 13913 raft_consensus.cc:504] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.651893 13566 catalog_manager.cc:5484] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.653355 13913 leader_election.cc:288] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.654911 13783 leader_election.cc:302] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:01.659276 13861 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.659744 13861 ts_tablet_manager.cc:1367] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.001s
I20240503 02:07:01.660027 13867 raft_consensus.cc:2781] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.660519 13867 raft_consensus.cc:686] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.661432 13867 consensus_queue.cc:237] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.662196 13861 raft_consensus.cc:348] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.662849 13861 raft_consensus.cc:374] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.663110 13861 raft_consensus.cc:729] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.663708 13861 consensus_queue.cc:260] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.660306 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.665488 13860 tablet_bootstrap.cc:654] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.666677 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:01.667204 13781 leader_election.cc:341] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:01.668002 13783 leader_election.cc:341] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.668429 13783 leader_election.cc:302] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.669164 13910 raft_consensus.cc:2726] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.670782 13861 ts_tablet_manager.cc:1398] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240503 02:07:01.671710 13861 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.675730 13858 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.676239 13858 ts_tablet_manager.cc:1367] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.003s
I20240503 02:07:01.676981 13860 tablet_bootstrap.cc:492] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.677407 13860 ts_tablet_manager.cc:1367] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20240503 02:07:01.679764 13860 raft_consensus.cc:348] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.680444 13860 raft_consensus.cc:374] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.680698 13860 raft_consensus.cc:729] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.681377 13860 consensus_queue.cc:260] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.681870 13858 raft_consensus.cc:348] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.682502 13858 raft_consensus.cc:374] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.682758 13858 raft_consensus.cc:729] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.683290 13860 ts_tablet_manager.cc:1398] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:01.683355 13858 consensus_queue.cc:260] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.684232 13860 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.689415 13858 ts_tablet_manager.cc:1398] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 02:07:01.689922 13860 tablet_bootstrap.cc:654] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.690286 13858 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.693032 13565 catalog_manager.cc:5484] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.694422 13565 catalog_manager.cc:5484] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.695816 13858 tablet_bootstrap.cc:654] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.688673 13861 tablet_bootstrap.cc:654] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.703666 13860 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.704164 13860 ts_tablet_manager.cc:1367] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20240503 02:07:01.706614 13860 raft_consensus.cc:348] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.707343 13860 raft_consensus.cc:374] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.707605 13860 raft_consensus.cc:729] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.708194 13860 consensus_queue.cc:260] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.710255 13867 consensus_queue.cc:1035] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:01.712285 13860 ts_tablet_manager.cc:1398] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240503 02:07:01.714259 13860 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.718281 13858 tablet_bootstrap.cc:492] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.718722 13858 ts_tablet_manager.cc:1367] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.002s
I20240503 02:07:01.719632 13860 tablet_bootstrap.cc:654] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.722741 13883 consensus_queue.cc:1035] T fe9c8e98bdb04950a65590a69bf0ac19 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:01.732542 13921 raft_consensus.cc:482] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.733038 13921 raft_consensus.cc:504] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.734936 13921 leader_election.cc:288] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.737345 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.737980 13818 raft_consensus.cc:2454] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:01.738552 13861 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.738423 13858 raft_consensus.cc:348] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.739076 13861 ts_tablet_manager.cc:1367] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.003s
I20240503 02:07:01.739457 13710 leader_election.cc:302] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.740389 13921 raft_consensus.cc:2781] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.740798 13921 raft_consensus.cc:482] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.739122 13858 raft_consensus.cc:374] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.741479 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.742070 13858 raft_consensus.cc:729] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.742453 13913 raft_consensus.cc:482] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.742720 13858 consensus_queue.cc:260] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.742949 13913 raft_consensus.cc:504] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.745046 13913 leader_election.cc:288] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.742465 13921 raft_consensus.cc:3037] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.745751 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893a056b25c462d82b4147b937c8c94" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.746523 13667 raft_consensus.cc:2454] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.746671 13858 ts_tablet_manager.cc:1398] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.028s user 0.004s sys 0.003s
I20240503 02:07:01.747753 13783 leader_election.cc:302] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.748143 13858 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.748888 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893a056b25c462d82b4147b937c8c94" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:01.750393 13781 leader_election.cc:341] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.751363 13913 raft_consensus.cc:2781] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.751744 13913 raft_consensus.cc:482] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.752066 13913 raft_consensus.cc:3037] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.753875 13858 tablet_bootstrap.cc:654] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.758607 13860 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.759068 13860 ts_tablet_manager.cc:1367] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.022s
I20240503 02:07:01.759374 13858 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.759035 13913 raft_consensus.cc:504] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.759794 13858 ts_tablet_manager.cc:1367] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 02:07:01.761546 13860 raft_consensus.cc:348] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.762269 13860 raft_consensus.cc:374] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.762089 13858 raft_consensus.cc:348] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.762440 13669 raft_consensus.cc:2454] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:01.762812 13858 raft_consensus.cc:374] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.762595 13860 raft_consensus.cc:729] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.763159 13858 raft_consensus.cc:729] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.763784 13858 consensus_queue.cc:260] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.763737 13860 consensus_queue.cc:260] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.765786 13858 ts_tablet_manager.cc:1398] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:07:01.767179 13921 raft_consensus.cc:504] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.768920 13860 ts_tablet_manager.cc:1398] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:07:01.769881 13860 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.769739 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.770335 13858 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.741837 13861 raft_consensus.cc:348] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.770758 13861 raft_consensus.cc:374] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.771040 13861 raft_consensus.cc:729] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.770288 13669 raft_consensus.cc:3037] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.775759 13860 tablet_bootstrap.cc:654] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.775877 13858 tablet_bootstrap.cc:654] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.777073 13861 consensus_queue.cc:260] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.782985 13858 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.783439 13858 ts_tablet_manager.cc:1367] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240503 02:07:01.785710 13858 raft_consensus.cc:348] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.786373 13858 raft_consensus.cc:374] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.786618 13858 raft_consensus.cc:729] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.786804 13860 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.787227 13860 ts_tablet_manager.cc:1367] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240503 02:07:01.787269 13858 consensus_queue.cc:260] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.788199 13861 ts_tablet_manager.cc:1398] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.049s user 0.005s sys 0.001s
I20240503 02:07:01.789997 13858 ts_tablet_manager.cc:1398] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:01.789718 13860 raft_consensus.cc:348] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.790395 13860 raft_consensus.cc:374] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.790668 13860 raft_consensus.cc:729] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.790841 13858 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.791363 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893a056b25c462d82b4147b937c8c94" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.791368 13860 consensus_queue.cc:260] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.792049 13667 raft_consensus.cc:3037] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.796663 13858 tablet_bootstrap.cc:654] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.797387 13860 ts_tablet_manager.cc:1398] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:07:01.798255 13860 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.798802 13667 raft_consensus.cc:2454] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.799885 13783 leader_election.cc:302] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.800802 13883 raft_consensus.cc:2781] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.801296 13883 raft_consensus.cc:686] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.802101 13883 consensus_queue.cc:237] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.803369 13860 tablet_bootstrap.cc:654] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.805749 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893a056b25c462d82b4147b937c8c94" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.806804 13744 raft_consensus.cc:3037] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.811059 13858 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.811575 13858 ts_tablet_manager.cc:1367] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20240503 02:07:01.812724 13669 raft_consensus.cc:2454] T 330da4b1ff064e5f865ec87c4b081cce P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:01.813767 13921 leader_election.cc:288] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.813905 13858 raft_consensus.cc:348] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.814831 13858 raft_consensus.cc:374] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.814888 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330da4b1ff064e5f865ec87c4b081cce" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:01.815330 13858 raft_consensus.cc:729] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.815543 13818 raft_consensus.cc:3037] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.816247 13858 consensus_queue.cc:260] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.818351 13858 ts_tablet_manager.cc:1398] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:01.821384 13861 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.822041 13858 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.824884 13565 catalog_manager.cc:5484] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.827428 13861 tablet_bootstrap.cc:654] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.827852 13913 leader_election.cc:288] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.828114 13858 tablet_bootstrap.cc:654] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.829375 13709 leader_election.cc:302] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:01.834483 13865 raft_consensus.cc:2781] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.834599 13744 raft_consensus.cc:2454] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.835059 13865 raft_consensus.cc:686] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:01.835889 13865 consensus_queue.cc:237] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.836521 13918 raft_consensus.cc:482] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.837013 13918 raft_consensus.cc:504] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.838991 13918 leader_election.cc:288] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.840204 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.841107 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:01.841598 13710 leader_election.cc:341] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.841799 13860 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.842252 13860 ts_tablet_manager.cc:1367] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.016s
I20240503 02:07:01.843034 13818 raft_consensus.cc:2454] T 330da4b1ff064e5f865ec87c4b081cce P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:01.844602 13860 raft_consensus.cc:348] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.845326 13860 raft_consensus.cc:374] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.845602 13860 raft_consensus.cc:729] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
W20240503 02:07:01.846231 13709 leader_election.cc:341] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.846359 13860 consensus_queue.cc:260] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.846699 13709 leader_election.cc:302] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.849385 13860 ts_tablet_manager.cc:1398] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:07:01.850382 13860 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.855434 13861 tablet_bootstrap.cc:492] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.855876 13861 ts_tablet_manager.cc:1367] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.007s
I20240503 02:07:01.856010 13860 tablet_bootstrap.cc:654] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.850386 13918 raft_consensus.cc:2726] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.859054 13861 raft_consensus.cc:348] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.859769 13861 raft_consensus.cc:374] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.860034 13861 raft_consensus.cc:729] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.860877 13861 consensus_queue.cc:260] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.866560 13913 raft_consensus.cc:482] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.867103 13913 raft_consensus.cc:504] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.868851 13858 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.869370 13858 ts_tablet_manager.cc:1367] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.013s
I20240503 02:07:01.869851 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.870106 13913 leader_election.cc:288] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.871055 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.871515 13744 raft_consensus.cc:2454] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.872076 13858 raft_consensus.cc:348] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.872449 13781 leader_election.cc:302] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.869961 13567 catalog_manager.cc:5484] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:01.872849 13858 raft_consensus.cc:374] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.873311 13858 raft_consensus.cc:729] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.873312 13913 raft_consensus.cc:2781] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.873911 13913 raft_consensus.cc:482] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.874250 13913 raft_consensus.cc:3037] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.874296 13858 consensus_queue.cc:260] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:01.876104 13783 leader_election.cc:341] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.877297 13858 ts_tablet_manager.cc:1398] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240503 02:07:01.878060 13858 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.879051 13861 ts_tablet_manager.cc:1398] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20240503 02:07:01.881496 13860 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:01.881922 13860 ts_tablet_manager.cc:1367] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.007s
I20240503 02:07:01.882613 13858 tablet_bootstrap.cc:654] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.885564 13913 raft_consensus.cc:504] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.888259 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:01.888456 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.889070 13744 raft_consensus.cc:3037] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:01.890290 13783 leader_election.cc:341] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.891853 13861 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:01.892879 13913 leader_election.cc:288] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.884269 13860 raft_consensus.cc:348] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.893891 13860 raft_consensus.cc:374] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.894243 13860 raft_consensus.cc:729] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.894994 13860 consensus_queue.cc:260] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.895593 13744 raft_consensus.cc:2454] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:01.896724 13781 leader_election.cc:302] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.897665 13861 tablet_bootstrap.cc:654] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.897872 13883 raft_consensus.cc:2781] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:01.898355 13883 raft_consensus.cc:686] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:01.899147 13883 consensus_queue.cc:237] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.901978 13860 ts_tablet_manager.cc:1398] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240503 02:07:01.902474 13923 raft_consensus.cc:482] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.903110 13860 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:01.902938 13923 raft_consensus.cc:504] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.907248 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:01.908859 13783 leader_election.cc:341] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.909271 13865 raft_consensus.cc:482] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.909720 13865 raft_consensus.cc:504] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.910315 13858 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:01.910753 13858 ts_tablet_manager.cc:1367] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.018s
I20240503 02:07:01.911049 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.911687 13743 raft_consensus.cc:2454] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.911794 13865 leader_election.cc:288] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.912577 13669 raft_consensus.cc:3037] T c4a20af2756040cea6c15f6ace3912bb P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.913305 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.914005 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.914413 13781 leader_election.cc:302] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.914640 13818 raft_consensus.cc:2454] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:01.913975 13667 raft_consensus.cc:2454] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:01.915650 13860 tablet_bootstrap.cc:654] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.915957 13710 leader_election.cc:302] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:01.913177 13858 raft_consensus.cc:348] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.916496 13858 raft_consensus.cc:374] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.916867 13858 raft_consensus.cc:729] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.917150 13866 raft_consensus.cc:482] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.917629 13865 raft_consensus.cc:2781] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.917598 13866 raft_consensus.cc:504] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.917891 13858 consensus_queue.cc:260] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.918207 13865 raft_consensus.cc:482] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.924734 13865 raft_consensus.cc:3037] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.909749 13923 leader_election.cc:288] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.926553 13923 consensus_queue.cc:1035] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:01.933606 13951 consensus_queue.cc:1035] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:01.919831 13858 ts_tablet_manager.cc:1398] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:07:01.921496 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:01.937476 13743 raft_consensus.cc:2454] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:01.922353 13910 raft_consensus.cc:2781] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.922516 13861 tablet_bootstrap.cc:492] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:01.938277 13910 raft_consensus.cc:482] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.938520 13861 ts_tablet_manager.cc:1367] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.010s
I20240503 02:07:01.938632 13910 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.941362 13858 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:01.924245 13867 raft_consensus.cc:482] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.942849 13867 raft_consensus.cc:504] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.923442 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:01.941680 13632 leader_election.cc:302] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:01.919616 13866 leader_election.cc:288] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.944983 13867 leader_election.cc:288] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:01.946321 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:01.947407 13866 raft_consensus.cc:2781] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.947754 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:01.948336 13783 leader_election.cc:341] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.948474 13742 raft_consensus.cc:2454] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:01.947782 13866 raft_consensus.cc:482] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.949352 13861 raft_consensus.cc:348] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.949782 13866 raft_consensus.cc:3037] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.950137 13861 raft_consensus.cc:374] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:01.950436 13861 raft_consensus.cc:729] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:01.945295 13818 raft_consensus.cc:2454] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:01.951649 13861 consensus_queue.cc:260] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.958182 13910 raft_consensus.cc:504] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.954803 13781 leader_election.cc:302] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:01.959761 13567 catalog_manager.cc:5484] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:01.961918 13923 raft_consensus.cc:2781] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:01.962325 13923 raft_consensus.cc:482] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:01.962671 13923 raft_consensus.cc:3037] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.963507 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
W20240503 02:07:01.964907 13783 leader_election.cc:341] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.966274 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.965126 13883 consensus_queue.cc:1035] T c4a20af2756040cea6c15f6ace3912bb P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:07:01.966934 13744 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.972867 13918 raft_consensus.cc:482] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:01.973371 13918 raft_consensus.cc:504] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.975292 13918 leader_election.cc:288] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:01.977360 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:01.978865 13709 leader_election.cc:341] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.941890 13865 raft_consensus.cc:504] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.980430 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:01.981815 13710 leader_election.cc:341] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:01.982275 13710 leader_election.cc:302] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:01.983693 13918 raft_consensus.cc:2726] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:01.959877 13866 raft_consensus.cc:504] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:01.986907 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:01.987534 13741 raft_consensus.cc:3037] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:01.994653 13741 raft_consensus.cc:2454] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:01.995662 13632 leader_election.cc:302] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:01.997015 13918 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.001611 13858 tablet_bootstrap.cc:654] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:01.966910 13910 leader_election.cc:288] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.003574 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.004253 13667 raft_consensus.cc:3037] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.005558 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "707d1ec02f89489092029bf2eee1761d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.006178 13816 raft_consensus.cc:3037] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.035271 13668 raft_consensus.cc:3037] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.037705 13667 raft_consensus.cc:2454] T 707d1ec02f89489092029bf2eee1761d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:02.039134 13866 leader_election.cc:288] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.040928 13709 leader_election.cc:302] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:02.041285 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e8e9e6dec94158989853c4dbb2787b" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.041977 13817 raft_consensus.cc:3037] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.042151 13918 raft_consensus.cc:2781] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.042670 13918 raft_consensus.cc:686] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:02.043236 13866 raft_consensus.cc:2781] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.043519 13918 consensus_queue.cc:237] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.021554 13865 leader_election.cc:288] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:02.053509 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:02.058166 13817 raft_consensus.cc:2454] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:02.012151 13744 raft_consensus.cc:2454] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:02.060431 13910 consensus_queue.cc:1035] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 02:07:02.061249 13866 raft_consensus.cc:686] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:02.069028 13866 consensus_queue.cc:237] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:02.066064 13632 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 70212.3us
I20240503 02:07:02.081518 13955 raft_consensus.cc:482] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.082005 13955 raft_consensus.cc:504] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.082533 13816 raft_consensus.cc:2454] T 707d1ec02f89489092029bf2eee1761d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:02.083873 13781 leader_election.cc:302] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.087049 13865 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.087384 13861 ts_tablet_manager.cc:1398] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.149s user 0.028s sys 0.079s
I20240503 02:07:02.089493 13861 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.088560 13867 raft_consensus.cc:2781] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.077358 13923 raft_consensus.cc:504] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.097303 13858 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.097813 13858 ts_tablet_manager.cc:1367] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.007s
I20240503 02:07:02.098712 13883 consensus_queue.cc:1035] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.100881 13955 leader_election.cc:288] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.101773 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.097702 13867 raft_consensus.cc:686] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:02.106666 13867 consensus_queue.cc:237] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.104136 13952 consensus_queue.cc:1035] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:07:02.100989 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.105851 13860 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.117623 13860 ts_tablet_manager.cc:1367] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.215s user 0.054s sys 0.056s
I20240503 02:07:02.101872 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed43100e614122b8bbc0bc3506d664" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.122159 13858 raft_consensus.cc:348] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.189126 13858 raft_consensus.cc:374] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.189435 13858 raft_consensus.cc:729] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.189992 13858 consensus_queue.cc:260] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.187876 13957 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.192277 13957 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.192708 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.193341 13744 raft_consensus.cc:3037] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.194308 13957 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.123265 13635 leader_election.cc:302] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.197088 13910 consensus_queue.cc:1035] T c4f6dafbb5974de589d4064a5f979f01 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.197402 13860 raft_consensus.cc:348] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.156816 13921 raft_consensus.cc:482] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.198083 13860 raft_consensus.cc:374] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.198309 13957 raft_consensus.cc:2726] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:07:02.198374 13860 raft_consensus.cc:729] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
W20240503 02:07:02.199278 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 76104.1us
I20240503 02:07:02.199230 13860 consensus_queue.cc:260] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.200542 13921 raft_consensus.cc:504] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.200807 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.184948 13923 leader_election.cc:288] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.201845 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.202733 13567 catalog_manager.cc:5484] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.156021 13865 raft_consensus.cc:482] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.205137 13858 ts_tablet_manager.cc:1398] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.107s user 0.018s sys 0.020s
I20240503 02:07:02.205488 13865 raft_consensus.cc:504] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.206003 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.136018 13955 raft_consensus.cc:482] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.207365 13858 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.208216 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c782825ce834414baad8c3f6b6642e5d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.208704 13921 leader_election.cc:288] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.209220 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.209622 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:02.125522 13783 leader_election.cc:341] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.154568 13952 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.211753 13860 ts_tablet_manager.cc:1398] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.094s user 0.012s sys 0.008s
I20240503 02:07:02.212396 13860 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.211326 13865 leader_election.cc:288] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:02.215260 13632 leader_election.cc:341] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.215351 13817 raft_consensus.cc:2454] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:02.217309 13952 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.215829 13710 leader_election.cc:302] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.218307 13818 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.218465 13865 raft_consensus.cc:2726] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.218891 13710 leader_election.cc:302] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.219102 13744 raft_consensus.cc:2454] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:02.219622 13865 raft_consensus.cc:2781] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.219950 13952 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.219967 13865 raft_consensus.cc:482] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.220772 13865 raft_consensus.cc:3037] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.220661 13781 leader_election.cc:302] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.211498 13955 raft_consensus.cc:504] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.225003 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.226074 13667 raft_consensus.cc:2454] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
W20240503 02:07:02.226447 13781 leader_election.cc:341] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.227341 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:02.214756 13783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 87574.6us
I20240503 02:07:02.229094 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.229689 13817 raft_consensus.cc:2454] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.227348 13955 leader_election.cc:288] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.230629 13952 raft_consensus.cc:2781] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.234187 13635 leader_election.cc:302] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.236470 13952 raft_consensus.cc:686] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:02.235369 13666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.238858 13666 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:02.240306 13955 raft_consensus.cc:2781] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.238279 13952 consensus_queue.cc:237] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.254491 13861 tablet_bootstrap.cc:654] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.254351 13783 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:02.240464 13921 consensus_queue.cc:1035] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240503 02:07:02.255373 13955 raft_consensus.cc:482] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.256100 13867 raft_consensus.cc:2781] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.247838 13865 raft_consensus.cc:504] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.257339 13867 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.256628 13955 raft_consensus.cc:3037] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.260766 13635 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:02.263921 13866 raft_consensus.cc:2781] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.264289 13866 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.264596 13866 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.266003 13744 raft_consensus.cc:3037] T c782825ce834414baad8c3f6b6642e5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.258409 13867 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.271003 13913 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.271481 13866 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.272135 13867 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.274358 13913 consensus_queue.cc:1035] T c782825ce834414baad8c3f6b6642e5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.277065 13742 raft_consensus.cc:2379] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:02.278390 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.284998 13818 raft_consensus.cc:3037] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.278635 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.281316 13858 tablet_bootstrap.cc:654] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:02.292004 13632 leader_election.cc:341] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.302665 13860 tablet_bootstrap.cc:654] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.300781 13665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.277099 13865 leader_election.cc:288] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.301129 13866 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.302290 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.302476 13867 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.316555 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.317072 13817 raft_consensus.cc:2379] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
W20240503 02:07:02.329017 13781 leader_election.cc:341] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.334441 13955 raft_consensus.cc:504] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.336449 13955 leader_election.cc:288] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.339084 13635 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.343128 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.343569 13866 raft_consensus.cc:2726] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.352075 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.353030 13665 raft_consensus.cc:2379] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.355612 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294969cac87345fa9dec0a1ed3b214f6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.358053 13664 raft_consensus.cc:2379] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b518e4e142f54c789a56606b766c4e06 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.363066 13818 raft_consensus.cc:2454] T 294969cac87345fa9dec0a1ed3b214f6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:02.363776 13743 raft_consensus.cc:2379] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:02.364914 13632 leader_election.cc:302] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.365048 13710 leader_election.cc:302] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.365402 13783 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.366024 13866 raft_consensus.cc:2726] T 294969cac87345fa9dec0a1ed3b214f6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.366369 13913 raft_consensus.cc:2726] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.364209 13858 tablet_bootstrap.cc:492] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.367121 13858 ts_tablet_manager.cc:1367] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.160s user 0.017s sys 0.003s
I20240503 02:07:02.369637 13971 raft_consensus.cc:2781] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.371452 13971 raft_consensus.cc:686] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:02.372521 13971 consensus_queue.cc:237] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.373929 13861 tablet_bootstrap.cc:492] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.374459 13861 ts_tablet_manager.cc:1367] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.286s user 0.041s sys 0.007s
I20240503 02:07:02.385195 13858 raft_consensus.cc:348] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.386080 13858 raft_consensus.cc:374] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.386354 13858 raft_consensus.cc:729] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.382169 13564 catalog_manager.cc:5484] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:02.394564 13858 consensus_queue.cc:260] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.396870 13858 ts_tablet_manager.cc:1398] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20240503 02:07:02.398196 13858 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.398267 13952 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:02.397505 13564 catalog_manager.cc:5484] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:02.401711 13970 consensus_queue.cc:1035] T 330da4b1ff064e5f865ec87c4b081cce P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:02.402213 13860 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.402688 13860 ts_tablet_manager.cc:1367] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.190s user 0.005s sys 0.006s
I20240503 02:07:02.405090 13860 raft_consensus.cc:348] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.406702 13860 raft_consensus.cc:374] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.407001 13860 raft_consensus.cc:729] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.407740 13860 consensus_queue.cc:260] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.412986 13858 tablet_bootstrap.cc:654] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.414911 13860 ts_tablet_manager.cc:1398] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240503 02:07:02.415896 13860 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.421214 13861 raft_consensus.cc:348] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.422003 13861 raft_consensus.cc:374] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.422140 13860 tablet_bootstrap.cc:654] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.422302 13861 raft_consensus.cc:729] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.422626 13566 catalog_manager.cc:5484] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.424561 13861 consensus_queue.cc:260] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.426837 13861 ts_tablet_manager.cc:1398] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.052s user 0.003s sys 0.001s
I20240503 02:07:02.427788 13861 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.430550 13913 raft_consensus.cc:482] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.431010 13913 raft_consensus.cc:504] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.432920 13913 leader_election.cc:288] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.434471 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.435243 13861 tablet_bootstrap.cc:654] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.435405 13860 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.435803 13860 ts_tablet_manager.cc:1367] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
W20240503 02:07:02.436559 13783 consensus_peers.cc:485] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 -> Peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Couldn't send request to peer 7da64b3eda0a40b6b183c1cfc19db738. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:07:02.435809 13781 leader_election.cc:341] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.438282 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.438377 13860 raft_consensus.cc:348] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.439136 13860 raft_consensus.cc:374] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.439407 13860 raft_consensus.cc:729] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
W20240503 02:07:02.439467 13783 leader_election.cc:341] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.439957 13783 leader_election.cc:302] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.440263 13860 consensus_queue.cc:260] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.441129 13913 raft_consensus.cc:2726] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.442816 13860 ts_tablet_manager.cc:1398] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:07:02.444308 13860 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.446666 13861 tablet_bootstrap.cc:492] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.447132 13861 ts_tablet_manager.cc:1367] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20240503 02:07:02.449391 13860 tablet_bootstrap.cc:654] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.457832 13861 raft_consensus.cc:348] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.458631 13861 raft_consensus.cc:374] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.458916 13861 raft_consensus.cc:729] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.460635 13860 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.459710 13861 consensus_queue.cc:260] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.461179 13860 ts_tablet_manager.cc:1367] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20240503 02:07:02.462626 13861 ts_tablet_manager.cc:1398] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240503 02:07:02.463469 13861 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.463438 13860 raft_consensus.cc:348] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.464217 13860 raft_consensus.cc:374] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.464473 13860 raft_consensus.cc:729] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.466835 13866 raft_consensus.cc:482] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.465230 13860 consensus_queue.cc:260] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.469005 13866 raft_consensus.cc:504] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.470844 13860 ts_tablet_manager.cc:1398] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:07:02.471693 13860 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.471644 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.472522 13742 raft_consensus.cc:2379] T 26b581675616419d8d97c4e8973455f4 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:02.476127 13913 raft_consensus.cc:482] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.476620 13913 raft_consensus.cc:504] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.478085 13866 leader_election.cc:288] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.478595 13913 leader_election.cc:288] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.478803 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b581675616419d8d97c4e8973455f4" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.479624 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.480327 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:02.486639 13783 leader_election.cc:341] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.486783 13635 leader_election.cc:302] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.482007 13860 tablet_bootstrap.cc:654] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:02.487639 13781 leader_election.cc:341] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.488063 13781 leader_election.cc:302] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.489398 13952 raft_consensus.cc:2726] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.490160 13866 raft_consensus.cc:3037] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.492612 13952 consensus_queue.cc:1035] T 26b581675616419d8d97c4e8973455f4 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.496675 13866 raft_consensus.cc:2726] T 26b581675616419d8d97c4e8973455f4 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.482007 13861 tablet_bootstrap.cc:654] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.502205 13564 catalog_manager.cc:5484] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.503355 13861 tablet_bootstrap.cc:492] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.510262 13861 ts_tablet_manager.cc:1367] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.001s
I20240503 02:07:02.512691 13861 raft_consensus.cc:348] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.513422 13861 raft_consensus.cc:374] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.513698 13861 raft_consensus.cc:729] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.514475 13861 consensus_queue.cc:260] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.516688 13861 ts_tablet_manager.cc:1398] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:07:02.519793 13861 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.523312 13860 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.523784 13860 ts_tablet_manager.cc:1367] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.009s
I20240503 02:07:02.523900 13861 tablet_bootstrap.cc:654] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.526239 13860 raft_consensus.cc:348] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.526943 13860 raft_consensus.cc:374] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.527218 13860 raft_consensus.cc:729] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.528862 13858 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.529301 13858 ts_tablet_manager.cc:1367] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.132s user 0.045s sys 0.031s
I20240503 02:07:02.532243 13858 raft_consensus.cc:348] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.532900 13861 tablet_bootstrap.cc:492] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.532943 13858 raft_consensus.cc:374] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.533375 13861 ts_tablet_manager.cc:1367] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20240503 02:07:02.533669 13860 consensus_queue.cc:260] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.534562 13858 raft_consensus.cc:729] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.535619 13858 consensus_queue.cc:260] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.535728 13861 raft_consensus.cc:348] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.536458 13861 raft_consensus.cc:374] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.536715 13861 raft_consensus.cc:729] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.537494 13861 consensus_queue.cc:260] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.539489 13861 ts_tablet_manager.cc:1398] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:02.537952 13858 ts_tablet_manager.cc:1398] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:07:02.540407 13861 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.539561 13860 ts_tablet_manager.cc:1398] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:07:02.541436 13860 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.540483 13858 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.545607 13952 raft_consensus.cc:482] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.545842 13861 tablet_bootstrap.cc:654] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.546108 13952 raft_consensus.cc:504] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.548022 13952 leader_election.cc:288] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.549410 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.549711 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:02.550860 13783 leader_election.cc:341] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:02.551458 13781 leader_election.cc:341] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.551875 13781 leader_election.cc:302] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.553021 13952 raft_consensus.cc:2726] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.554045 13858 tablet_bootstrap.cc:654] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.558275 13952 raft_consensus.cc:482] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.558730 13952 raft_consensus.cc:504] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.560014 13860 tablet_bootstrap.cc:654] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.560678 13952 leader_election.cc:288] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.558706 13861 tablet_bootstrap.cc:492] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.561520 13861 ts_tablet_manager.cc:1367] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20240503 02:07:02.561777 13970 consensus_queue.cc:1035] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.563803 13861 raft_consensus.cc:348] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.564261 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.564532 13861 raft_consensus.cc:374] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.564951 13861 raft_consensus.cc:729] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.564936 13744 raft_consensus.cc:2454] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:02.565645 13861 consensus_queue.cc:260] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.566184 13781 leader_election.cc:302] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.567000 13858 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.567229 13952 raft_consensus.cc:2781] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.567430 13858 ts_tablet_manager.cc:1367] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240503 02:07:02.567574 13861 ts_tablet_manager.cc:1398] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:02.567620 13952 raft_consensus.cc:482] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.568109 13952 raft_consensus.cc:3037] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.568594 13861 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.570760 13858 raft_consensus.cc:348] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.571408 13858 raft_consensus.cc:374] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.571673 13858 raft_consensus.cc:729] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.572263 13858 consensus_queue.cc:260] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.563844 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.574666 13861 tablet_bootstrap.cc:654] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:02.575131 13783 leader_election.cc:341] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.577210 13860 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.577646 13860 ts_tablet_manager.cc:1367] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.004s
I20240503 02:07:02.579991 13860 raft_consensus.cc:348] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.581012 13860 raft_consensus.cc:374] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.581274 13860 raft_consensus.cc:729] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.581810 13860 consensus_queue.cc:260] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.583534 13860 ts_tablet_manager.cc:1398] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:02.584318 13860 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.588317 13952 raft_consensus.cc:504] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.589391 13860 tablet_bootstrap.cc:654] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.590277 13952 leader_election.cc:288] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.590875 13866 raft_consensus.cc:482] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.591328 13866 raft_consensus.cc:504] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.592461 13861 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.592931 13861 ts_tablet_manager.cc:1367] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.004s
I20240503 02:07:02.593240 13866 leader_election.cc:288] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.594343 13858 ts_tablet_manager.cc:1398] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20240503 02:07:02.595752 13858 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.595527 13861 raft_consensus.cc:348] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.596218 13861 raft_consensus.cc:374] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.596499 13861 raft_consensus.cc:729] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.597131 13861 consensus_queue.cc:260] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.597436 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.599110 13861 ts_tablet_manager.cc:1398] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:02.599915 13861 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.601193 13858 tablet_bootstrap.cc:654] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.605703 13861 tablet_bootstrap.cc:654] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.606249 13860 tablet_bootstrap.cc:492] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.606675 13860 ts_tablet_manager.cc:1367] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240503 02:07:02.609050 13860 raft_consensus.cc:348] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.609745 13860 raft_consensus.cc:374] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.610005 13860 raft_consensus.cc:729] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.610428 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2524d54d0a2b4efc84ca69b6057c97ef" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.611232 13744 raft_consensus.cc:2379] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:02.611179 13971 consensus_queue.cc:1035] T 707d1ec02f89489092029bf2eee1761d P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:02.612795 13632 leader_election.cc:302] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.614023 13955 raft_consensus.cc:3037] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.615424 13858 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.615881 13858 ts_tablet_manager.cc:1367] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.002s
I20240503 02:07:02.618248 13858 raft_consensus.cc:348] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.618885 13858 raft_consensus.cc:374] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.619130 13858 raft_consensus.cc:729] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.619750 13858 consensus_queue.cc:260] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.610575 13860 consensus_queue.cc:260] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.622275 13860 ts_tablet_manager.cc:1398] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240503 02:07:02.622049 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.623150 13860 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.623840 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.625280 13744 raft_consensus.cc:3037] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:02.625701 13783 leader_election.cc:341] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.630143 13858 ts_tablet_manager.cc:1398] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240503 02:07:02.630923 13955 raft_consensus.cc:2726] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.639509 13866 raft_consensus.cc:482] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.639948 13866 raft_consensus.cc:504] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.631073 13858 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.632630 13971 raft_consensus.cc:482] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.641330 13971 raft_consensus.cc:504] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.643301 13971 leader_election.cc:288] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.643214 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.643898 13742 raft_consensus.cc:2454] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.645061 13632 leader_election.cc:302] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:02.645797 13866 leader_election.cc:288] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.646272 13866 raft_consensus.cc:2781] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.646567 13866 raft_consensus.cc:482] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.646876 13866 raft_consensus.cc:3037] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.647868 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.635959 13744 raft_consensus.cc:2454] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:02.648952 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.649570 13818 raft_consensus.cc:2454] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:02.650856 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.650935 13710 leader_election.cc:302] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.651479 13818 raft_consensus.cc:2454] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.651739 13971 raft_consensus.cc:2781] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.652092 13971 raft_consensus.cc:482] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.652115 13664 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.652390 13971 raft_consensus.cc:3037] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.653702 13781 leader_election.cc:302] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:02.654505 13952 raft_consensus.cc:2781] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.654943 13952 raft_consensus.cc:686] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:02.655666 13952 consensus_queue.cc:237] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.657107 13861 tablet_bootstrap.cc:492] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.657582 13861 ts_tablet_manager.cc:1367] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.019s
I20240503 02:07:02.659957 13861 raft_consensus.cc:348] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.660710 13861 raft_consensus.cc:374] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.661029 13861 raft_consensus.cc:729] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.661665 13861 consensus_queue.cc:260] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.662680 13910 consensus_queue.cc:1035] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.664237 13861 ts_tablet_manager.cc:1398] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:02.664640 13957 raft_consensus.cc:482] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.665318 13861 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.665139 13957 raft_consensus.cc:504] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.667140 13957 leader_election.cc:288] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.666875 13866 raft_consensus.cc:504] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.655686 13860 tablet_bootstrap.cc:654] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.670861 13861 tablet_bootstrap.cc:654] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.670825 13865 raft_consensus.cc:482] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.662307 13971 raft_consensus.cc:504] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.671468 13865 raft_consensus.cc:504] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.672148 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.672919 13742 raft_consensus.cc:2379] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:02.673310 13667 raft_consensus.cc:2379] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.673560 13858 tablet_bootstrap.cc:654] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.673804 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.674407 13744 raft_consensus.cc:2454] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:02.673388 13865 leader_election.cc:288] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.675545 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.677186 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.677754 13817 raft_consensus.cc:3037] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.679113 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.679944 13667 raft_consensus.cc:2379] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.680757 13971 leader_election.cc:288] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.681608 13632 leader_election.cc:302] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:02.677186 13665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:02.683558 13709 leader_election.cc:341] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.686050 13866 leader_election.cc:288] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.688876 13817 raft_consensus.cc:2454] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:02.693590 13866 raft_consensus.cc:2781] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:07:02.694689 13957 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:02.695079 13861 tablet_bootstrap.cc:492] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.695750 13861 ts_tablet_manager.cc:1367] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.004s
I20240503 02:07:02.698998 13957 raft_consensus.cc:482] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.699107 13866 raft_consensus.cc:482] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b518e4e142f54c789a56606b766c4e06)
I20240503 02:07:02.699595 13866 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:07:02.699463 13957 raft_consensus.cc:504] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.700078 13861 raft_consensus.cc:348] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.701015 13861 raft_consensus.cc:374] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.701318 13861 raft_consensus.cc:729] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.702215 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.702098 13861 consensus_queue.cc:260] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.702914 13742 raft_consensus.cc:2454] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.704231 13861 ts_tablet_manager.cc:1398] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:07:02.705621 13861 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.705798 13866 raft_consensus.cc:504] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.706854 13635 leader_election.cc:302] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:02.707626 13866 leader_election.cc:288] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.708400 13866 raft_consensus.cc:2781] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.710584 13858 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:02.704192 13632 leader_election.cc:302] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:02.711182 13923 raft_consensus.cc:482] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.711624 13923 raft_consensus.cc:504] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.712083 13858 ts_tablet_manager.cc:1367] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.001s
I20240503 02:07:02.712152 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.712771 13742 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:07:02.713523 13923 leader_election.cc:288] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.714955 13861 tablet_bootstrap.cc:654] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.714531 13858 raft_consensus.cc:348] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.717208 13858 raft_consensus.cc:374] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.716162 14005 raft_consensus.cc:2781] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.717788 14005 raft_consensus.cc:482] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.718008 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.718679 13667 raft_consensus.cc:2454] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:02.719408 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6854ff6c7c458c89a06fc412aca3f0" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.720221 13818 raft_consensus.cc:2379] T fb6854ff6c7c458c89a06fc412aca3f0 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.717533 13858 raft_consensus.cc:729] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.718190 14005 raft_consensus.cc:3037] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.721233 13866 raft_consensus.cc:686] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:02.721804 13858 consensus_queue.cc:260] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.722652 13923 consensus_queue.cc:1035] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.722080 13866 consensus_queue.cc:237] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.724401 13783 leader_election.cc:302] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.725173 13923 raft_consensus.cc:2781] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.725546 13923 raft_consensus.cc:482] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.725847 13923 raft_consensus.cc:3037] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.728425 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.729107 13741 raft_consensus.cc:2454] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:02.729566 13565 catalog_manager.cc:5484] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:02.733568 13710 leader_election.cc:302] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.734643 13971 raft_consensus.cc:2726] T fb6854ff6c7c458c89a06fc412aca3f0 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.736385 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cf17ea09b444b2afbc0d9a5bd38b79" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.736495 13971 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.739610 13971 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.741554 13971 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.743376 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.744138 13668 raft_consensus.cc:2379] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:02.744195 13957 leader_election.cc:288] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.745153 13858 ts_tablet_manager.cc:1398] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.033s user 0.002s sys 0.004s
I20240503 02:07:02.745337 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.746131 13858 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:02.746127 13816 raft_consensus.cc:2379] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:02.747601 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.753235 13710 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.754335 13971 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.758214 13910 consensus_queue.cc:1035] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:07:02.761334 13710 leader_election.cc:341] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.761819 13710 leader_election.cc:302] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.766649 13957 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.769280 13664 raft_consensus.cc:1226] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 FOLLOWER]: Rejecting Update request from peer b518e4e142f54c789a56606b766c4e06 for earlier term 1. Current term is 2. Ops: []
I20240503 02:07:02.780483 13860 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:02.783615 13955 raft_consensus.cc:482] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.792245 13955 raft_consensus.cc:504] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.794234 13955 leader_election.cc:288] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.740643 13742 raft_consensus.cc:2454] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 2.
I20240503 02:07:02.795398 13744 raft_consensus.cc:1226] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 2 FOLLOWER]: Rejecting Update request from peer b518e4e142f54c789a56606b766c4e06 for earlier term 1. Current term is 2. Ops: []
I20240503 02:07:02.760672 13971 raft_consensus.cc:2726] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.796133 13632 leader_election.cc:302] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.797022 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.798072 13955 raft_consensus.cc:2781] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:07:02.797936 13952 consensus_queue.cc:1046] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.798544 13955 raft_consensus.cc:686] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:02.798831 13952 raft_consensus.cc:3032] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:07:02.799187 13952 raft_consensus.cc:729] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:02.799419 13955 consensus_queue.cc:237] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.799969 13952 consensus_queue.cc:260] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.801136 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2493a234a8424ef6a6393a690a9f8f5d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.801899 13744 raft_consensus.cc:2379] T 2493a234a8424ef6a6393a690a9f8f5d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:02.803001 13632 leader_election.cc:302] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.804035 13955 raft_consensus.cc:3037] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.789271 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.805037 13816 raft_consensus.cc:2454] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.811388 13918 raft_consensus.cc:2726] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:07:02.839901 13710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 78598us
I20240503 02:07:02.796176 13860 ts_tablet_manager.cc:1367] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.173s user 0.013s sys 0.009s
I20240503 02:07:02.843297 13861 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.843783 13861 ts_tablet_manager.cc:1367] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.138s user 0.036s sys 0.047s
I20240503 02:07:02.844048 13858 tablet_bootstrap.cc:654] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.845769 13860 raft_consensus.cc:348] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.848064 13860 raft_consensus.cc:374] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.848472 13860 raft_consensus.cc:729] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.849304 13860 consensus_queue.cc:260] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.849521 13861 raft_consensus.cc:348] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.850286 13861 raft_consensus.cc:374] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:02.850435 13955 raft_consensus.cc:2726] T 2493a234a8424ef6a6393a690a9f8f5d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:02.850592 13861 raft_consensus.cc:729] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:02.851336 13861 consensus_queue.cc:260] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.853621 13861 ts_tablet_manager.cc:1398] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 02:07:02.854637 13861 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:02.839452 13952 raft_consensus.cc:3037] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:07:02.819547 13910 consensus_queue.cc:1159] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7da64b3eda0a40b6b183c1cfc19db738" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b518e4e142f54c789a56606b766c4e06 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240503 02:07:02.818413 13923 raft_consensus.cc:504] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.868148 13860 ts_tablet_manager.cc:1398] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240503 02:07:02.869087 13860 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:02.836236 14005 raft_consensus.cc:504] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.876665 14005 leader_election.cc:288] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.877743 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.877904 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60da30fcdc7f4fee9a61dd710d55673c" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:02.878376 13742 raft_consensus.cc:3037] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.878508 13816 raft_consensus.cc:3037] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.882053 13923 leader_election.cc:288] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:02.883090 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:02.883688 13744 raft_consensus.cc:3037] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.884202 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2a2e3ecd3f407ca6cd7a32ac82e89b" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:02.884900 13667 raft_consensus.cc:3037] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.888312 13861 tablet_bootstrap.cc:654] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.890822 13565 catalog_manager.cc:5484] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.893170 13918 consensus_queue.cc:1035] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:02.894116 13816 raft_consensus.cc:2454] T 60da30fcdc7f4fee9a61dd710d55673c P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:02.895231 13635 leader_election.cc:302] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.896311 14005 raft_consensus.cc:2781] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.896852 14005 raft_consensus.cc:686] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:02.896466 13744 raft_consensus.cc:2454] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:02.897706 14005 consensus_queue.cc:237] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.899044 13781 leader_election.cc:302] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:02.899832 13910 raft_consensus.cc:2781] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:02.900332 13910 raft_consensus.cc:686] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:02.901232 13910 consensus_queue.cc:237] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.903018 13565 catalog_manager.cc:5484] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 1 to 2, leader changed from b518e4e142f54c789a56606b766c4e06 (127.12.53.3) to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.896364 13742 raft_consensus.cc:2454] T 60da30fcdc7f4fee9a61dd710d55673c P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:02.900719 13667 raft_consensus.cc:2454] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:02.946132 13971 consensus_queue.cc:1035] T 294969cac87345fa9dec0a1ed3b214f6 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:02.951918 13918 raft_consensus.cc:482] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.952500 13918 raft_consensus.cc:504] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.952728 13565 catalog_manager.cc:5484] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:02.954603 13918 leader_election.cc:288] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.955137 13918 raft_consensus.cc:482] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.955544 13918 raft_consensus.cc:504] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.956957 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d610b8f0ef44e33880de53f13a9dd3e" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:02.957562 13667 raft_consensus.cc:2454] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:02.945339 14005 raft_consensus.cc:482] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:02.959736 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d610b8f0ef44e33880de53f13a9dd3e" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.959852 14005 raft_consensus.cc:504] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:02.965050 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:02.965653 13742 raft_consensus.cc:2454] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
W20240503 02:07:02.966455 13710 leader_election.cc:341] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:02.966872 13632 leader_election.cc:302] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:02.967666 14007 raft_consensus.cc:2781] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.968003 14007 raft_consensus.cc:482] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.968315 14007 raft_consensus.cc:3037] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.971846 14005 leader_election.cc:288] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.977564 13860 tablet_bootstrap.cc:654] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:02.984105 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.986109 13918 leader_election.cc:288] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:02.991804 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:02.993436 13816 raft_consensus.cc:2454] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:02.975538 13709 leader_election.cc:302] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:02.998010 13918 raft_consensus.cc:2781] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:02.998437 13918 raft_consensus.cc:482] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:02.998826 13918 raft_consensus.cc:3037] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:02.998044 13861 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:02.999755 13861 ts_tablet_manager.cc:1367] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.145s user 0.014s sys 0.010s
I20240503 02:07:03.000486 14007 raft_consensus.cc:504] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.003273 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.003479 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.002133 13861 raft_consensus.cc:348] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.004308 13818 raft_consensus.cc:3037] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.004496 13861 raft_consensus.cc:374] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.004869 13861 raft_consensus.cc:729] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.005487 13742 raft_consensus.cc:3037] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.006577 13861 consensus_queue.cc:260] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.005312 14007 leader_election.cc:288] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.010269 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab629d73c1c84dbbb3c7d3cd00dedf2a" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.011152 13664 raft_consensus.cc:2379] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:03.011232 13858 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.013794 13858 ts_tablet_manager.cc:1367] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.268s user 0.012s sys 0.006s
I20240503 02:07:03.016968 13858 raft_consensus.cc:348] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.017642 13858 raft_consensus.cc:374] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.017987 13858 raft_consensus.cc:729] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.013059 13709 leader_election.cc:302] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:03.018424 13742 raft_consensus.cc:2454] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.018962 13861 ts_tablet_manager.cc:1398] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20240503 02:07:03.020481 13861 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.020373 13632 leader_election.cc:302] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.021531 14007 raft_consensus.cc:2781] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.022079 14007 raft_consensus.cc:686] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:03.018548 13858 consensus_queue.cc:260] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.027225 13865 raft_consensus.cc:2726] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.028251 13858 ts_tablet_manager.cc:1398] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240503 02:07:03.029072 13858 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.018651 13918 raft_consensus.cc:504] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.035763 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d610b8f0ef44e33880de53f13a9dd3e" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:03.035923 14007 consensus_queue.cc:237] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.036409 13664 raft_consensus.cc:3037] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.042683 13664 raft_consensus.cc:2454] T 7d610b8f0ef44e33880de53f13a9dd3e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:03.044227 13818 raft_consensus.cc:2454] T ab629d73c1c84dbbb3c7d3cd00dedf2a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.044504 13860 tablet_bootstrap.cc:492] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.044975 13860 ts_tablet_manager.cc:1367] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.176s user 0.017s sys 0.003s
I20240503 02:07:03.045267 13709 leader_election.cc:302] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.049353 13860 raft_consensus.cc:348] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.050259 13860 raft_consensus.cc:374] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.051226 13860 raft_consensus.cc:729] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.051885 13860 consensus_queue.cc:260] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.052104 13918 leader_election.cc:288] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.054180 13860 ts_tablet_manager.cc:1398] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:07:03.055094 13860 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.057147 13971 raft_consensus.cc:2781] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.057852 13971 raft_consensus.cc:686] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:03.058394 13861 tablet_bootstrap.cc:654] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.058652 13971 consensus_queue.cc:237] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.060950 13860 tablet_bootstrap.cc:654] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.062881 13918 raft_consensus.cc:482] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.063194 13858 tablet_bootstrap.cc:654] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.063352 13918 raft_consensus.cc:504] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.065093 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d610b8f0ef44e33880de53f13a9dd3e" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.065709 13818 raft_consensus.cc:3037] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.066557 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.067039 13918 leader_election.cc:288] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:03.068121 13710 leader_election.cc:341] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.069118 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:03.071331 13709 leader_election.cc:341] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.071781 13709 leader_election.cc:302] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:03.072511 13918 raft_consensus.cc:2726] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.074391 13861 tablet_bootstrap.cc:492] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.074832 13861 ts_tablet_manager.cc:1367] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.002s
I20240503 02:07:03.074929 13860 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.075336 13860 ts_tablet_manager.cc:1367] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.006s
I20240503 02:07:03.077232 13861 raft_consensus.cc:348] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.077937 13861 raft_consensus.cc:374] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.077710 13860 raft_consensus.cc:348] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.078219 13861 raft_consensus.cc:729] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.078388 13860 raft_consensus.cc:374] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.078667 13860 raft_consensus.cc:729] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.078847 13861 consensus_queue.cc:260] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.079267 13860 consensus_queue.cc:260] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.080909 13861 ts_tablet_manager.cc:1398] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240503 02:07:03.081926 13860 ts_tablet_manager.cc:1398] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:07:03.083563 13818 raft_consensus.cc:2454] T 7d610b8f0ef44e33880de53f13a9dd3e P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:03.083861 13861 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.084138 13860 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.085697 13858 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.086128 13858 ts_tablet_manager.cc:1367] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.004s
I20240503 02:07:03.088482 13858 raft_consensus.cc:348] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.089216 13858 raft_consensus.cc:374] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.089468 13858 raft_consensus.cc:729] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.051040 13567 catalog_manager.cc:5484] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:03.090049 13860 tablet_bootstrap.cc:654] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.093835 13861 tablet_bootstrap.cc:654] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.095122 13860 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.095549 13860 ts_tablet_manager.cc:1367] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
W20240503 02:07:03.093781 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:03.090052 13858 consensus_queue.cc:260] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.097915 13860 raft_consensus.cc:348] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.098596 13860 raft_consensus.cc:374] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.098866 13860 raft_consensus.cc:729] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.099021 13861 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.099543 13861 ts_tablet_manager.cc:1367] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20240503 02:07:03.099915 13858 ts_tablet_manager.cc:1398] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240503 02:07:03.099483 13860 consensus_queue.cc:260] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.100926 13858 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.103430 13563 catalog_manager.cc:5484] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:03.111127 13860 ts_tablet_manager.cc:1398] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240503 02:07:03.146458 13858 tablet_bootstrap.cc:654] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.143424 13860 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.151474 13861 raft_consensus.cc:348] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.152220 13861 raft_consensus.cc:374] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.152493 13861 raft_consensus.cc:729] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.155871 13860 tablet_bootstrap.cc:654] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.155634 14005 consensus_queue.cc:1035] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.120841 13952 consensus_queue.cc:1035] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20240503 02:07:03.159371 13923 consensus_queue.cc:1035] T 2493a234a8424ef6a6393a690a9f8f5d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.160818 13861 consensus_queue.cc:260] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.118949 14007 raft_consensus.cc:482] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.170831 14007 raft_consensus.cc:504] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.176092 13866 consensus_queue.cc:1035] T fb6854ff6c7c458c89a06fc412aca3f0 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:03.177876 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.178638 13744 raft_consensus.cc:2454] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.180399 13632 leader_election.cc:302] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.181823 13866 raft_consensus.cc:2781] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.182229 13866 raft_consensus.cc:482] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.182567 13866 raft_consensus.cc:3037] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.193116 14007 leader_election.cc:288] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.195086 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.205296 13861 ts_tablet_manager.cc:1398] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.105s user 0.007s sys 0.000s
I20240503 02:07:03.206387 13861 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.219205 13866 raft_consensus.cc:504] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.223299 13567 catalog_manager.cc:5484] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:03.202664 13818 raft_consensus.cc:2454] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.231485 13866 leader_election.cc:288] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.241396 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.241535 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b722a96f2794d91bbd9163a7f283c27" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.244801 13818 raft_consensus.cc:3037] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.245515 13866 raft_consensus.cc:482] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.245963 13866 raft_consensus.cc:504] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.247859 13866 leader_election.cc:288] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.249141 13860 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.248920 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.249402 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.249635 13860 ts_tablet_manager.cc:1367] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.106s user 0.026s sys 0.049s
W20240503 02:07:03.250334 13635 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.244277 13741 raft_consensus.cc:3037] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:03.250834 13632 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.251272 13632 leader_election.cc:302] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:03.251981 13866 raft_consensus.cc:2726] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.259475 13860 raft_consensus.cc:348] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.260212 13860 raft_consensus.cc:374] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.260481 13860 raft_consensus.cc:729] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.260541 13818 raft_consensus.cc:2454] T 7b722a96f2794d91bbd9163a7f283c27 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.261242 13860 consensus_queue.cc:260] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.261618 13635 leader_election.cc:302] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.262600 13741 raft_consensus.cc:2454] T 7b722a96f2794d91bbd9163a7f283c27 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.264294 13860 ts_tablet_manager.cc:1398] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240503 02:07:03.265208 13860 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.263386 13866 raft_consensus.cc:2781] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.266212 13866 raft_consensus.cc:686] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:03.266990 13866 consensus_queue.cc:237] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.267995 13858 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.268427 13858 ts_tablet_manager.cc:1367] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.168s user 0.073s sys 0.041s
I20240503 02:07:03.269773 13861 tablet_bootstrap.cc:654] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.270941 13858 raft_consensus.cc:348] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.272433 13858 raft_consensus.cc:374] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.272634 13858 raft_consensus.cc:729] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.273252 13858 consensus_queue.cc:260] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.275311 13858 ts_tablet_manager.cc:1398] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:07:03.276182 13858 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.278596 13860 tablet_bootstrap.cc:654] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.284034 13858 tablet_bootstrap.cc:654] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.289094 13860 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.289710 13860 ts_tablet_manager.cc:1367] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.002s
I20240503 02:07:03.291954 13860 raft_consensus.cc:348] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.292670 13860 raft_consensus.cc:374] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.292994 13860 raft_consensus.cc:729] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.293686 13860 consensus_queue.cc:260] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.296576 13860 ts_tablet_manager.cc:1398] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:07:03.297664 13741 raft_consensus.cc:1261] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 65b596345545434b9f09935907294192 [term 2 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:07:03.298171 13858 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.298628 13858 ts_tablet_manager.cc:1367] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.000s
I20240503 02:07:03.299069 13866 consensus_queue.cc:1035] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.299614 13860 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.300928 13858 raft_consensus.cc:348] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.301599 13858 raft_consensus.cc:374] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.301849 13858 raft_consensus.cc:729] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.303535 13858 consensus_queue.cc:260] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.305089 13910 raft_consensus.cc:482] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.305477 13858 ts_tablet_manager.cc:1398] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:07:03.305547 13910 raft_consensus.cc:504] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.307108 13860 tablet_bootstrap.cc:654] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.307559 13910 leader_election.cc:288] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.309428 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a74eddb0bc474cb7c1e14749665244" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.310150 13742 raft_consensus.cc:2454] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:03.311429 13781 leader_election.cc:302] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.311972 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a74eddb0bc474cb7c1e14749665244" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
W20240503 02:07:03.313277 13783 leader_election.cc:341] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.309728 13858 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.316210 13910 raft_consensus.cc:2781] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.316634 13910 raft_consensus.cc:482] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.317001 13910 raft_consensus.cc:3037] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.319880 13818 raft_consensus.cc:1261] T c5cf17ea09b444b2afbc0d9a5bd38b79 P b518e4e142f54c789a56606b766c4e06 [term 2 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:07:03.322100 14007 consensus_queue.cc:1035] T c5cf17ea09b444b2afbc0d9a5bd38b79 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:03.327139 13858 tablet_bootstrap.cc:654] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.328804 13860 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.330261 13860 ts_tablet_manager.cc:1367] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.003s
I20240503 02:07:03.332818 13860 raft_consensus.cc:348] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.333681 13860 raft_consensus.cc:374] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.333942 13860 raft_consensus.cc:729] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.334537 13860 consensus_queue.cc:260] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.335795 13910 raft_consensus.cc:504] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.339224 13910 leader_election.cc:288] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.341742 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a74eddb0bc474cb7c1e14749665244" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.342443 13741 raft_consensus.cc:3037] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.342417 13861 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.343111 13861 ts_tablet_manager.cc:1367] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.137s user 0.021s sys 0.033s
I20240503 02:07:03.345645 13910 raft_consensus.cc:482] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.346938 13910 raft_consensus.cc:504] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.347572 13860 ts_tablet_manager.cc:1398] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:07:03.348445 13860 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.353502 13665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.341821 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a74eddb0bc474cb7c1e14749665244" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
W20240503 02:07:03.356398 13783 leader_election.cc:341] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.352576 13566 catalog_manager.cc:5484] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
W20240503 02:07:03.357112 13783 leader_election.cc:341] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.358428 13918 raft_consensus.cc:482] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.358884 13918 raft_consensus.cc:504] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.363384 13860 tablet_bootstrap.cc:654] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.363955 13918 leader_election.cc:288] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.369539 13866 consensus_queue.cc:1035] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:03.371448 13858 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.371589 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.372045 13858 ts_tablet_manager.cc:1367] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.001s
I20240503 02:07:03.372344 13668 raft_consensus.cc:2454] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:03.373513 13709 leader_election.cc:302] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.374321 13918 raft_consensus.cc:2781] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.374665 13918 raft_consensus.cc:482] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.374981 13918 raft_consensus.cc:3037] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.374965 13858 raft_consensus.cc:348] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.375895 13858 raft_consensus.cc:374] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.376221 13858 raft_consensus.cc:729] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.376616 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.377353 13818 raft_consensus.cc:2454] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:03.377064 13858 consensus_queue.cc:260] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.381382 13918 raft_consensus.cc:504] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.382367 13858 ts_tablet_manager.cc:1398] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20240503 02:07:03.383837 13858 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.383872 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:03.384518 13668 raft_consensus.cc:3037] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.387240 14005 consensus_queue.cc:1035] T 60da30fcdc7f4fee9a61dd710d55673c P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.390081 13858 tablet_bootstrap.cc:654] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.390863 13668 raft_consensus.cc:2454] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:03.391949 13709 leader_election.cc:302] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.392720 13971 raft_consensus.cc:2781] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.393194 13971 raft_consensus.cc:686] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:03.394026 13971 consensus_queue.cc:237] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.395145 13741 raft_consensus.cc:2454] T 78a74eddb0bc474cb7c1e14749665244 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:03.396806 13910 leader_election.cc:288] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.397578 13781 leader_election.cc:302] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:03.399024 13910 raft_consensus.cc:2781] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.399503 13910 raft_consensus.cc:686] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:03.399909 13918 leader_election.cc:288] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.373050 13861 raft_consensus.cc:348] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.400287 13910 consensus_queue.cc:237] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.400964 13861 raft_consensus.cc:374] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.401276 13861 raft_consensus.cc:729] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.395834 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07abdb29bc6d463c8928a23e8579ebac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.402196 13817 raft_consensus.cc:3037] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.401876 13861 consensus_queue.cc:260] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.403509 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:03.404798 13781 leader_election.cc:341] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.405391 13861 ts_tablet_manager.cc:1398] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.062s user 0.009s sys 0.000s
I20240503 02:07:03.406297 13861 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.405253 13781 leader_election.cc:302] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:03.409734 13952 raft_consensus.cc:2726] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.412055 13861 tablet_bootstrap.cc:654] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.420181 13860 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.419267 13817 raft_consensus.cc:2454] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:03.420650 13860 ts_tablet_manager.cc:1367] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.072s user 0.019s sys 0.009s
I20240503 02:07:03.423173 13866 raft_consensus.cc:482] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.423636 13866 raft_consensus.cc:504] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.426882 13866 leader_election.cc:288] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.428769 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.429430 13818 raft_consensus.cc:2454] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.430701 13635 leader_election.cc:302] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.431353 13860 raft_consensus.cc:348] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.432058 13860 raft_consensus.cc:374] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.432317 13860 raft_consensus.cc:729] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.432433 14005 raft_consensus.cc:2781] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.432636 13861 tablet_bootstrap.cc:492] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.432814 14005 raft_consensus.cc:482] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.433321 14005 raft_consensus.cc:3037] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.433398 13861 ts_tablet_manager.cc:1367] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.004s
I20240503 02:07:03.435853 13860 consensus_queue.cc:260] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.436231 13861 raft_consensus.cc:348] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.437281 13861 raft_consensus.cc:374] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.437742 13861 raft_consensus.cc:729] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.438449 13861 consensus_queue.cc:260] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.440286 13860 ts_tablet_manager.cc:1398] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240503 02:07:03.440718 13861 ts_tablet_manager.cc:1398] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 02:07:03.440474 14005 raft_consensus.cc:504] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.442325 13861 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.442759 13858 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.443370 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.443951 13858 ts_tablet_manager.cc:1367] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.060s user 0.014s sys 0.018s
W20240503 02:07:03.444865 13632 leader_election.cc:341] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.446260 13858 raft_consensus.cc:348] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.446943 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.447575 13818 raft_consensus.cc:3037] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.448819 13861 tablet_bootstrap.cc:654] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.454180 13818 raft_consensus.cc:2454] T b15197927046474b804042923de03d31 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.455401 13635 leader_election.cc:302] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:03.456241 13866 raft_consensus.cc:2781] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.456804 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.456724 13866 raft_consensus.cc:686] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
W20240503 02:07:03.458071 13632 leader_election.cc:341] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.457994 13866 consensus_queue.cc:237] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.458801 13563 catalog_manager.cc:5484] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:03.460624 13921 raft_consensus.cc:482] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.442711 13860 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.461115 13921 raft_consensus.cc:504] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.456259 14005 leader_election.cc:288] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.463028 13921 leader_election.cc:288] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.464787 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.466850 13860 tablet_bootstrap.cc:654] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:03.470094 13709 leader_election.cc:341] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.472409 13913 consensus_queue.cc:1035] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.478411 13921 consensus_queue.cc:1035] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.486685 13910 raft_consensus.cc:482] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.487190 13910 raft_consensus.cc:504] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.489228 13910 leader_election.cc:288] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.486047 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.490944 13861 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.490623 13566 catalog_manager.cc:5484] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
W20240503 02:07:03.491400 13710 leader_election.cc:341] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.446954 13858 raft_consensus.cc:374] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.491848 13710 leader_election.cc:302] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:03.492180 13858 raft_consensus.cc:729] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.493046 13971 raft_consensus.cc:2726] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.491384 13861 ts_tablet_manager.cc:1367] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.012s
I20240503 02:07:03.496551 13861 raft_consensus.cc:348] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.497272 13861 raft_consensus.cc:374] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.492895 13858 consensus_queue.cc:260] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.497519 13861 raft_consensus.cc:729] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.498134 13861 consensus_queue.cc:260] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.498950 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b89ae3ce565499ebcaab8912fedc385" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.500149 13861 ts_tablet_manager.cc:1398] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20240503 02:07:03.500375 13781 leader_election.cc:341] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.500773 13858 ts_tablet_manager.cc:1398] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20240503 02:07:03.501060 13861 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.501690 13858 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.502780 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b89ae3ce565499ebcaab8912fedc385" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.503402 13667 raft_consensus.cc:2454] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:03.508808 13858 tablet_bootstrap.cc:654] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.509958 13783 leader_election.cc:302] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:03.511937 13923 raft_consensus.cc:2781] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.512341 13923 raft_consensus.cc:482] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.512667 13923 raft_consensus.cc:3037] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.519459 13861 tablet_bootstrap.cc:654] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.508355 13565 catalog_manager.cc:5484] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:03.522820 13952 consensus_queue.cc:1035] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.527871 13923 raft_consensus.cc:504] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.532868 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b89ae3ce565499ebcaab8912fedc385" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:03.533458 13664 raft_consensus.cc:3037] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.533885 13861 tablet_bootstrap.cc:492] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.534320 13861 ts_tablet_manager.cc:1367] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.002s
I20240503 02:07:03.536612 13861 raft_consensus.cc:348] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.537330 13861 raft_consensus.cc:374] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.537575 13861 raft_consensus.cc:729] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.538110 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b89ae3ce565499ebcaab8912fedc385" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.538199 13923 leader_election.cc:288] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
W20240503 02:07:03.540292 13781 leader_election.cc:341] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.543226 13861 consensus_queue.cc:260] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.548318 13861 ts_tablet_manager.cc:1398] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20240503 02:07:03.549288 13861 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.554283 13861 tablet_bootstrap.cc:654] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.557715 13866 raft_consensus.cc:482] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.558282 13866 raft_consensus.cc:504] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.560256 13866 leader_election.cc:288] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.562270 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.563005 13817 raft_consensus.cc:2454] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.564424 13635 leader_election.cc:302] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.565569 13955 raft_consensus.cc:2781] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.565948 13955 raft_consensus.cc:482] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.566287 13955 raft_consensus.cc:3037] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.568121 13861 tablet_bootstrap.cc:492] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.568559 13861 ts_tablet_manager.cc:1367] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240503 02:07:03.571082 13866 consensus_queue.cc:1035] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.572402 13861 raft_consensus.cc:348] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.573132 13861 raft_consensus.cc:374] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.576818 13861 raft_consensus.cc:729] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.577467 13861 consensus_queue.cc:260] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.578766 13664 raft_consensus.cc:2454] T 4b89ae3ce565499ebcaab8912fedc385 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:03.580626 13783 leader_election.cc:302] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:03.581430 13861 ts_tablet_manager.cc:1398] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240503 02:07:03.581605 13923 raft_consensus.cc:2781] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.582306 13861 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.582522 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.583221 13744 raft_consensus.cc:2454] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.584995 13923 raft_consensus.cc:686] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:03.585532 13955 raft_consensus.cc:504] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.585855 13923 consensus_queue.cc:237] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.587514 13955 leader_election.cc:288] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.587715 13861 tablet_bootstrap.cc:654] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.589839 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.589869 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4904ea1d27d64f01a7ce69d4efa093af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.590970 13818 raft_consensus.cc:3037] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.592573 13744 raft_consensus.cc:3037] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.596417 13952 raft_consensus.cc:482] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.602250 13952 raft_consensus.cc:504] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.605783 13952 leader_election.cc:288] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.607960 13860 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.608479 13860 ts_tablet_manager.cc:1367] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.166s user 0.019s sys 0.034s
I20240503 02:07:03.609236 13744 raft_consensus.cc:2454] T 4904ea1d27d64f01a7ce69d4efa093af P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.609387 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85f42f870e749bb9cd2aecf025d3a20" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.610039 13741 raft_consensus.cc:2454] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:03.611009 13860 raft_consensus.cc:348] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.611559 13860 raft_consensus.cc:374] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.611723 13860 raft_consensus.cc:729] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.613518 13860 consensus_queue.cc:260] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.614601 13955 raft_consensus.cc:482] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.617384 13866 consensus_queue.cc:1035] T ab629d73c1c84dbbb3c7d3cd00dedf2a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240503 02:07:03.620149 13781 leader_election.cc:302] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.624843 13952 raft_consensus.cc:2781] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.625195 13952 raft_consensus.cc:482] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.625510 13952 raft_consensus.cc:3037] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.632972 13923 raft_consensus.cc:482] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.621843 13632 leader_election.cc:302] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.633555 13923 raft_consensus.cc:504] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.634356 14005 raft_consensus.cc:2781] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.634878 14005 raft_consensus.cc:686] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:03.635639 14005 consensus_queue.cc:237] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.627941 13563 catalog_manager.cc:5484] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:03.619690 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85f42f870e749bb9cd2aecf025d3a20" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.650817 13860 ts_tablet_manager.cc:1398] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.042s user 0.009s sys 0.002s
I20240503 02:07:03.651875 13860 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.649389 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:03.659335 13861 tablet_bootstrap.cc:492] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.659862 13861 ts_tablet_manager.cc:1367] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.004s
I20240503 02:07:03.660889 13858 tablet_bootstrap.cc:492] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.661255 13858 ts_tablet_manager.cc:1367] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.160s user 0.038s sys 0.016s
W20240503 02:07:03.655413 13783 leader_election.cc:341] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:03.664418 13861 raft_consensus.cc:348] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.666607 13861 raft_consensus.cc:374] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.666970 13861 raft_consensus.cc:729] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.665866 13858 raft_consensus.cc:348] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.667739 13858 raft_consensus.cc:374] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.668074 13858 raft_consensus.cc:729] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.669461 13858 consensus_queue.cc:260] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.649423 13664 raft_consensus.cc:2454] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:03.674386 13818 raft_consensus.cc:2454] T 4904ea1d27d64f01a7ce69d4efa093af P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.673971 13952 raft_consensus.cc:504] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.675280 13923 leader_election.cc:288] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.670418 13861 consensus_queue.cc:260] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.676605 13952 leader_election.cc:288] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.620126 13955 raft_consensus.cc:504] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.678359 13955 leader_election.cc:288] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.672475 13858 ts_tablet_manager.cc:1398] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20240503 02:07:03.681011 13858 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.681335 13861 ts_tablet_manager.cc:1398] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.021s user 0.007s sys 0.001s
I20240503 02:07:03.682185 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85f42f870e749bb9cd2aecf025d3a20" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.682790 13744 raft_consensus.cc:3037] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.683343 13861 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.685118 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.685745 13743 raft_consensus.cc:2454] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.685904 13858 tablet_bootstrap.cc:654] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.689800 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.690456 13817 raft_consensus.cc:2454] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.691910 13860 tablet_bootstrap.cc:654] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.693619 13632 leader_election.cc:302] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.695318 13861 tablet_bootstrap.cc:654] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.696610 14007 raft_consensus.cc:2781] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.697135 14007 raft_consensus.cc:482] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.697470 14007 raft_consensus.cc:3037] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.704800 13866 raft_consensus.cc:482] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.705314 13866 raft_consensus.cc:504] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.707180 13866 leader_election.cc:288] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.709806 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.710494 13743 raft_consensus.cc:2454] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.710394 13565 catalog_manager.cc:5484] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:03.711700 13632 leader_election.cc:302] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:03.713270 14005 raft_consensus.cc:2781] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.692102 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85f42f870e749bb9cd2aecf025d3a20" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:03.714376 13664 raft_consensus.cc:3037] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.714627 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.715230 13817 raft_consensus.cc:2454] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:03.718081 14007 raft_consensus.cc:504] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.713632 14005 raft_consensus.cc:482] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.725077 14005 raft_consensus.cc:3037] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.736191 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.738008 13742 raft_consensus.cc:2454] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:03.739149 13781 leader_election.cc:302] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:03.740360 13952 raft_consensus.cc:2781] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:03.740767 13952 raft_consensus.cc:482] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.741106 13952 raft_consensus.cc:3037] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.745265 13744 raft_consensus.cc:2454] T d85f42f870e749bb9cd2aecf025d3a20 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:03.746505 13781 leader_election.cc:302] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.747270 13923 raft_consensus.cc:2781] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.747738 13923 raft_consensus.cc:686] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:03.748512 13923 consensus_queue.cc:237] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.751461 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.752094 13744 raft_consensus.cc:3037] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.760721 14007 leader_election.cc:288] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.766872 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1251d1d9cfc6460ca25952c2cfa21817" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.767522 13817 raft_consensus.cc:3037] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.724722 13664 raft_consensus.cc:2454] T d85f42f870e749bb9cd2aecf025d3a20 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:03.787019 13918 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.803407 13817 raft_consensus.cc:2454] T 1251d1d9cfc6460ca25952c2cfa21817 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.804445 13635 leader_election.cc:302] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.805300 13955 raft_consensus.cc:2781] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.805771 13955 raft_consensus.cc:686] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:03.806527 13955 consensus_queue.cc:237] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.815433 13957 consensus_queue.cc:1035] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.817972 13860 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.818615 13860 ts_tablet_manager.cc:1367] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.167s user 0.042s sys 0.027s
I20240503 02:07:03.820657 14005 raft_consensus.cc:504] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.821902 13860 raft_consensus.cc:348] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.823339 13952 raft_consensus.cc:504] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.824717 13861 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.824694 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:03.825798 13861 ts_tablet_manager.cc:1367] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.143s user 0.026s sys 0.043s
I20240503 02:07:03.825899 13741 raft_consensus.cc:3037] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:03.828048 13861 raft_consensus.cc:348] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.824914 13860 raft_consensus.cc:374] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.834106 13860 raft_consensus.cc:729] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.834981 13860 consensus_queue.cc:260] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.828701 13861 raft_consensus.cc:374] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.836975 13861 raft_consensus.cc:729] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.837855 13861 consensus_queue.cc:260] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.839005 13860 ts_tablet_manager.cc:1398] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240503 02:07:03.842152 13952 leader_election.cc:288] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:03.843093 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:03.839643 13957 consensus_queue.cc:1035] T 7b722a96f2794d91bbd9163a7f283c27 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.843968 13668 raft_consensus.cc:2379] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b518e4e142f54c789a56606b766c4e06 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:03.844064 13860 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:03.842939 13861 ts_tablet_manager.cc:1398] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20240503 02:07:03.850564 13861 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:03.855955 13861 tablet_bootstrap.cc:654] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.864041 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
W20240503 02:07:03.864483 13783 consensus_peers.cc:485] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 -> Peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Couldn't send request to peer 7da64b3eda0a40b6b183c1cfc19db738. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:03.869752 13860 tablet_bootstrap.cc:654] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.873598 13781 leader_election.cc:302] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:03.875898 13913 raft_consensus.cc:2726] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:07:03.877820 14005 leader_election.cc:288] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.890383 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19c9d349b744a759025ccddcf3633a6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:03.891777 14005 consensus_queue.cc:1035] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.892726 13815 raft_consensus.cc:2379] T b19c9d349b744a759025ccddcf3633a6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:03.895958 14005 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.895902 13741 raft_consensus.cc:2454] T b19c9d349b744a759025ccddcf3633a6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.898955 13971 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:03.906612 14005 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.908016 13744 raft_consensus.cc:2454] T 1251d1d9cfc6460ca25952c2cfa21817 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.912195 13858 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:03.912770 13858 ts_tablet_manager.cc:1367] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.232s user 0.083s sys 0.094s
I20240503 02:07:03.913245 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:03.913964 13742 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.914377 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:03.915020 13814 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:03.915663 13858 raft_consensus.cc:348] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.916143 13635 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:03.916424 13858 raft_consensus.cc:374] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.916822 13858 raft_consensus.cc:729] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.916951 14005 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.917201 13955 raft_consensus.cc:2781] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:07:03.917644 13955 raft_consensus.cc:482] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:03.918255 13955 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:07:03.917673 13858 consensus_queue.cc:260] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.928231 13563 catalog_manager.cc:5484] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:03.935424 13858 ts_tablet_manager.cc:1398] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240503 02:07:03.936574 13858 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:03.935519 13632 leader_election.cc:302] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:03.944953 13957 raft_consensus.cc:2781] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:03.945449 13957 raft_consensus.cc:686] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:03.946215 13957 consensus_queue.cc:237] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.947916 13860 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:03.953244 13860 ts_tablet_manager.cc:1367] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.008s
I20240503 02:07:03.959450 13923 consensus_queue.cc:1035] T 78a74eddb0bc474cb7c1e14749665244 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.961622 13971 consensus_queue.cc:1035] T 7d610b8f0ef44e33880de53f13a9dd3e P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:03.964354 13861 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:03.964893 13861 ts_tablet_manager.cc:1367] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.115s user 0.011s sys 0.005s
I20240503 02:07:03.967370 13861 raft_consensus.cc:348] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.969089 13861 raft_consensus.cc:374] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:03.969420 13861 raft_consensus.cc:729] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:03.976261 13861 consensus_queue.cc:260] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.981133 13971 raft_consensus.cc:482] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:03.990535 13971 raft_consensus.cc:504] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:03.997108 13971 leader_election.cc:288] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:03.997673 13858 tablet_bootstrap.cc:654] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:03.997812 13861 ts_tablet_manager.cc:1398] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240503 02:07:03.998705 13861 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.005235 13955 raft_consensus.cc:504] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.005566 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:04.007128 13710 leader_election.cc:341] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.011065 13860 raft_consensus.cc:348] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.012094 13860 raft_consensus.cc:374] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.012339 13860 raft_consensus.cc:729] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.009800 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.012976 13921 raft_consensus.cc:482] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.024098 13921 raft_consensus.cc:504] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.025228 13742 raft_consensus.cc:3037] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.026343 13921 leader_election.cc:288] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.028587 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.029770 13955 leader_election.cc:288] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.029892 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:04.030627 13743 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:07:04.030676 13709 leader_election.cc:341] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.031306 13709 leader_election.cc:302] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:04.042315 13921 raft_consensus.cc:2726] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.077023 13913 raft_consensus.cc:482] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.081130 13913 raft_consensus.cc:504] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.110050 14007 raft_consensus.cc:482] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.110657 14007 raft_consensus.cc:504] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.112538 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.111337 13913 leader_election.cc:288] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
W20240503 02:07:04.116878 13781 leader_election.cc:341] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.119797 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:04.121313 13632 leader_election.cc:341] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.124377 14007 leader_election.cc:288] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.141726 13858 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:04.142302 13858 ts_tablet_manager.cc:1367] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.206s user 0.032s sys 0.012s
I20240503 02:07:04.143172 13861 tablet_bootstrap.cc:654] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.144940 13858 raft_consensus.cc:348] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.145084 13860 consensus_queue.cc:260] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.145697 13858 raft_consensus.cc:374] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.145995 13858 raft_consensus.cc:729] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.144973 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15197927046474b804042923de03d31" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:04.146623 13858 consensus_queue.cc:260] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.147392 13860 ts_tablet_manager.cc:1398] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.194s user 0.011s sys 0.007s
I20240503 02:07:04.148121 13858 ts_tablet_manager.cc:1398] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:04.148262 13860 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:04.149051 13858 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:04.154789 13860 tablet_bootstrap.cc:654] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.154834 13858 tablet_bootstrap.cc:654] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:04.171272 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:04.186543 13743 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 2.
I20240503 02:07:04.211483 13861 tablet_bootstrap.cc:492] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.235267 13566 catalog_manager.cc:5484] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:04.250813 13861 ts_tablet_manager.cc:1367] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.252s user 0.045s sys 0.007s
I20240503 02:07:04.253327 13861 raft_consensus.cc:348] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.254040 13861 raft_consensus.cc:374] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.254318 13861 raft_consensus.cc:729] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.255322 13861 consensus_queue.cc:260] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.255797 13632 leader_election.cc:302] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:04.257129 13861 ts_tablet_manager.cc:1398] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 02:07:04.257560 14075 raft_consensus.cc:2781] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:07:04.258031 13861 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.258020 14075 raft_consensus.cc:686] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [term 2 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:04.258890 14075 consensus_queue.cc:237] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.266767 13861 tablet_bootstrap.cc:654] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.278156 13861 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.278595 13861 ts_tablet_manager.cc:1367] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20240503 02:07:04.280278 14075 consensus_queue.cc:1035] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.281296 13861 raft_consensus.cc:348] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.282032 13861 raft_consensus.cc:374] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.282300 13861 raft_consensus.cc:729] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.282852 13861 consensus_queue.cc:260] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.285686 13861 ts_tablet_manager.cc:1398] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 02:07:04.286522 13861 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.291572 13861 tablet_bootstrap.cc:654] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.300370 13971 raft_consensus.cc:482] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.300989 13971 raft_consensus.cc:504] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.303083 13971 leader_election.cc:288] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.304967 13860 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:04.305454 13860 ts_tablet_manager.cc:1367] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.157s user 0.024s sys 0.019s
I20240503 02:07:04.308346 13860 raft_consensus.cc:348] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.309336 13860 raft_consensus.cc:374] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.309604 13860 raft_consensus.cc:729] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.310189 13860 consensus_queue.cc:260] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.310753 13861 tablet_bootstrap.cc:492] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.311007 14075 consensus_queue.cc:1035] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.311381 13861 ts_tablet_manager.cc:1367] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.005s
I20240503 02:07:04.314509 13860 ts_tablet_manager.cc:1398] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:07:04.314388 13861 raft_consensus.cc:348] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.315320 13861 raft_consensus.cc:374] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.315423 13860 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:04.315688 13861 raft_consensus.cc:729] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.316426 13861 consensus_queue.cc:260] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.318773 13861 ts_tablet_manager.cc:1398] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:07:04.319620 13861 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.315465 13957 consensus_queue.cc:1035] T b15197927046474b804042923de03d31 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:04.322046 13860 tablet_bootstrap.cc:654] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.325560 13861 tablet_bootstrap.cc:654] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.328192 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.328933 13668 raft_consensus.cc:2454] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:04.332672 13709 leader_election.cc:302] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:04.333539 13921 raft_consensus.cc:2781] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:04.333895 13921 raft_consensus.cc:482] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:04.334287 13921 raft_consensus.cc:3037] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.341706 13921 raft_consensus.cc:504] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.342504 13710 leader_election.cc:302] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:04.345890 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7c74febdf447aebc8b991b45c4e60a" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:04.346509 13812 raft_consensus.cc:3037] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:07:04.343493 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:04.362156 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:04.362946 13815 raft_consensus.cc:2454] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:04.362195 13667 raft_consensus.cc:3037] T 78a74eddb0bc474cb7c1e14749665244 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.367769 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.371958 13812 raft_consensus.cc:2454] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 2.
I20240503 02:07:04.374471 13635 leader_election.cc:302] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:04.376336 14075 raft_consensus.cc:2781] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:04.376796 14075 raft_consensus.cc:482] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:04.377171 14075 raft_consensus.cc:3037] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.381625 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:04.382261 13668 raft_consensus.cc:3037] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:04.386170 13710 leader_election.cc:341] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.391103 13921 leader_election.cc:288] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.397449 13668 raft_consensus.cc:2454] T 67b296978fa84ad6a6f9cc7ec6d202ac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:04.398581 13709 leader_election.cc:302] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:04.399420 13918 raft_consensus.cc:2781] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:04.399873 13918 raft_consensus.cc:686] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:04.400684 13918 consensus_queue.cc:237] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:04.407327 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 58850.3us
I20240503 02:07:04.409559 13957 raft_consensus.cc:482] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.410097 13957 raft_consensus.cc:504] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.416437 13910 raft_consensus.cc:482] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.417145 13910 raft_consensus.cc:504] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.419155 13910 leader_election.cc:288] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:04.419627 14005 consensus_queue.cc:1035] T 4904ea1d27d64f01a7ce69d4efa093af P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.419194 14007 consensus_queue.cc:1035] T 1251d1d9cfc6460ca25952c2cfa21817 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:07:04.422905 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
W20240503 02:07:04.424547 13710 leader_election.cc:341] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.427078 13971 raft_consensus.cc:2726] T b15197927046474b804042923de03d31 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.428978 13858 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:04.429551 13858 ts_tablet_manager.cc:1367] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.281s user 0.028s sys 0.039s
I20240503 02:07:04.432386 14075 raft_consensus.cc:504] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.445266 13957 leader_election.cc:288] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.452529 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.453256 13739 raft_consensus.cc:2454] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:04.454804 13858 raft_consensus.cc:348] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.455515 13858 raft_consensus.cc:374] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.455797 13858 raft_consensus.cc:729] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.459661 13858 consensus_queue.cc:260] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.462842 13858 ts_tablet_manager.cc:1398] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.033s user 0.005s sys 0.002s
I20240503 02:07:04.463797 13858 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:04.443673 14075 leader_election.cc:288] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.470165 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:04.470762 13815 raft_consensus.cc:3037] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.475419 13952 consensus_queue.cc:1035] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:07:04.475948 13783 leader_election.cc:341] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.476399 13783 leader_election.cc:302] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:04.476711 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:04.468678 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d633c024e481d9dcd7dc94259a65d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:04.483664 13739 raft_consensus.cc:3037] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.475451 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.489233 13861 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.489709 13861 ts_tablet_manager.cc:1367] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.170s user 0.062s sys 0.056s
I20240503 02:07:04.492115 13861 raft_consensus.cc:348] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.492827 13861 raft_consensus.cc:374] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.493116 13861 raft_consensus.cc:729] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.497345 13913 raft_consensus.cc:2726] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.499565 13815 raft_consensus.cc:2454] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:04.499828 13666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.493788 13861 consensus_queue.cc:260] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.501554 13635 leader_election.cc:302] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:04.508661 13957 raft_consensus.cc:2781] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:04.509200 13957 raft_consensus.cc:686] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:04.509992 13957 consensus_queue.cc:237] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.510445 13566 catalog_manager.cc:5484] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 2 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:04.512383 13566 catalog_manager.cc:5484] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
W20240503 02:07:04.514621 13781 consensus_peers.cc:485] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 -> Peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Couldn't send request to peer 65b596345545434b9f09935907294192. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:04.514837 13632 leader_election.cc:302] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:04.514861 13861 ts_tablet_manager.cc:1398] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20240503 02:07:04.516147 13861 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.502979 13817 raft_consensus.cc:2454] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:04.516669 14078 raft_consensus.cc:2781] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:04.518205 14078 raft_consensus.cc:482] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:04.518556 14078 raft_consensus.cc:3037] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.522347 13860 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:04.522814 13860 ts_tablet_manager.cc:1367] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.208s user 0.005s sys 0.011s
I20240503 02:07:04.526898 13860 raft_consensus.cc:348] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.527588 13860 raft_consensus.cc:374] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.528033 13860 raft_consensus.cc:729] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.529359 13860 consensus_queue.cc:260] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.553766 13860 ts_tablet_manager.cc:1398] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.031s user 0.005s sys 0.003s
I20240503 02:07:04.539685 14078 raft_consensus.cc:504] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.557678 13910 consensus_queue.cc:1035] T d85f42f870e749bb9cd2aecf025d3a20 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240503 02:07:04.563346 14078 leader_election.cc:288] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.542790 13739 raft_consensus.cc:2454] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:04.564983 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:04.568486 13861 tablet_bootstrap.cc:654] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.569092 13815 raft_consensus.cc:3037] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.571064 13858 tablet_bootstrap.cc:654] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.571327 13957 consensus_queue.cc:1035] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.582736 13815 raft_consensus.cc:2454] T b1ac22299784411188965b2bfbd51ef6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
W20240503 02:07:04.582937 13783 leader_election.cc:341] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.585328 13635 leader_election.cc:302] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:04.585224 13565 catalog_manager.cc:5484] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:04.586179 14094 consensus_queue.cc:1035] T 4b89ae3ce565499ebcaab8912fedc385 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240503 02:07:04.586529 14078 raft_consensus.cc:2781] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:04.586977 14078 raft_consensus.cc:686] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:04.587755 14078 consensus_queue.cc:237] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:04.589392 13781 leader_election.cc:341] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.589854 13781 leader_election.cc:302] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:04.591746 13952 raft_consensus.cc:2726] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.593926 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ac22299784411188965b2bfbd51ef6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:04.594570 13741 raft_consensus.cc:3037] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.625582 13860 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:04.626293 13858 tablet_bootstrap.cc:492] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:04.626685 13858 ts_tablet_manager.cc:1367] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.163s user 0.010s sys 0.001s
I20240503 02:07:04.647248 13741 raft_consensus.cc:2454] T b1ac22299784411188965b2bfbd51ef6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:04.676772 13858 raft_consensus.cc:348] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.677810 13858 raft_consensus.cc:374] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.678120 13858 raft_consensus.cc:729] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.678944 13858 consensus_queue.cc:260] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:04.680343 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 89229.5us
I20240503 02:07:04.683847 14093 raft_consensus.cc:482] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.685873 14093 raft_consensus.cc:504] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.689332 14093 leader_election.cc:288] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:04.689718 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:04.691403 13858 ts_tablet_manager.cc:1398] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20240503 02:07:04.692252 13858 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:04.695823 13860 tablet_bootstrap.cc:654] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:04.696566 13783 leader_election.cc:341] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.706781 13957 consensus_queue.cc:1035] T b19c9d349b744a759025ccddcf3633a6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.707635 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.710054 13861 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.710528 13861 ts_tablet_manager.cc:1367] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.195s user 0.027s sys 0.029s
W20240503 02:07:04.709321 13781 leader_election.cc:341] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.716260 13781 leader_election.cc:302] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:04.717341 13910 raft_consensus.cc:2726] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.719429 13861 raft_consensus.cc:348] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.720244 13861 raft_consensus.cc:374] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.720512 13861 raft_consensus.cc:729] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.728987 13861 consensus_queue.cc:260] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.744143 13565 catalog_manager.cc:5484] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:04.745282 13861 ts_tablet_manager.cc:1398] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.034s user 0.002s sys 0.014s
I20240503 02:07:04.746675 13565 catalog_manager.cc:5484] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:04.750452 13818 raft_consensus.cc:1261] T ee7c74febdf447aebc8b991b45c4e60a P b518e4e142f54c789a56606b766c4e06 [term 2 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:07:04.752955 13957 consensus_queue.cc:1035] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.760586 13861 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:04.773347 13957 raft_consensus.cc:482] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.773882 13957 raft_consensus.cc:504] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.773622 13861 tablet_bootstrap.cc:654] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.786809 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:04.788043 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.785355 13957 leader_election.cc:288] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.827769 13858 tablet_bootstrap.cc:654] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:04.832580 13710 consensus_peers.cc:485] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 -> Peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Couldn't send request to peer b518e4e142f54c789a56606b766c4e06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:04.833112 13860 tablet_bootstrap.cc:492] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:04.833631 13860 ts_tablet_manager.cc:1367] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.208s user 0.025s sys 0.024s
W20240503 02:07:04.833995 13635 leader_election.cc:341] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.836021 13860 raft_consensus.cc:348] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.836709 13860 raft_consensus.cc:374] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.837103 13860 raft_consensus.cc:729] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.837702 13860 consensus_queue.cc:260] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.839704 13860 ts_tablet_manager.cc:1398] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20240503 02:07:04.843092 13632 leader_election.cc:341] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.846691 13860 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:04.853920 13739 raft_consensus.cc:3037] T 4b89ae3ce565499ebcaab8912fedc385 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:04.857281 13741 raft_consensus.cc:1261] T ee7c74febdf447aebc8b991b45c4e60a P 65b596345545434b9f09935907294192 [term 2 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:07:04.871978 13860 tablet_bootstrap.cc:654] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.843575 13632 leader_election.cc:302] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:04.876044 14075 raft_consensus.cc:2726] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.878007 13858 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:04.878463 13858 ts_tablet_manager.cc:1367] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.186s user 0.017s sys 0.012s
I20240503 02:07:04.881471 13858 raft_consensus.cc:348] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.882220 13858 raft_consensus.cc:374] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.882483 13858 raft_consensus.cc:729] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.883126 13858 consensus_queue.cc:260] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.885192 13858 ts_tablet_manager.cc:1398] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:07:04.886031 13858 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:04.890511 13860 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:04.890976 13860 ts_tablet_manager.cc:1367] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.001s
I20240503 02:07:04.891731 13858 tablet_bootstrap.cc:654] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.892019 14007 consensus_queue.cc:1035] T ee7c74febdf447aebc8b991b45c4e60a P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.893229 13860 raft_consensus.cc:348] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.893903 13860 raft_consensus.cc:374] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.894160 13860 raft_consensus.cc:729] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.894737 13860 consensus_queue.cc:260] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.911877 13860 ts_tablet_manager.cc:1398] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.021s user 0.007s sys 0.001s
I20240503 02:07:04.912858 13860 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:04.919961 14007 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.922911 13957 raft_consensus.cc:482] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:04.923413 13957 raft_consensus.cc:504] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.925308 13957 leader_election.cc:288] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:04.926357 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:04.928628 13635 leader_election.cc:341] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.931749 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:04.932417 13858 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:04.934186 13860 tablet_bootstrap.cc:654] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.934100 13858 ts_tablet_manager.cc:1367] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.048s user 0.025s sys 0.010s
W20240503 02:07:04.933198 13632 leader_election.cc:341] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:04.939327 13632 leader_election.cc:302] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:04.940068 13957 raft_consensus.cc:2726] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:04.945201 14078 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.941476 13858 raft_consensus.cc:348] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.962218 13858 raft_consensus.cc:374] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.962483 13858 raft_consensus.cc:729] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.963050 13858 consensus_queue.cc:260] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.966408 13858 ts_tablet_manager.cc:1398] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20240503 02:07:04.967299 13858 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:04.973443 13861 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:04.973121 13921 consensus_queue.cc:1035] T 67b296978fa84ad6a6f9cc7ec6d202ac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:04.974152 13858 tablet_bootstrap.cc:654] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:04.978719 13860 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:04.979210 13860 ts_tablet_manager.cc:1367] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.013s
I20240503 02:07:04.982414 13860 raft_consensus.cc:348] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.983096 13860 raft_consensus.cc:374] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:04.983355 13860 raft_consensus.cc:729] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:04.984228 13861 ts_tablet_manager.cc:1367] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.224s user 0.022s sys 0.016s
I20240503 02:07:04.990109 13861 raft_consensus.cc:348] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.985277 13860 consensus_queue.cc:260] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:04.993266 13860 ts_tablet_manager.cc:1398] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20240503 02:07:04.994275 13860 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.007896 13860 tablet_bootstrap.cc:654] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.008241 13861 raft_consensus.cc:374] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.008647 13861 raft_consensus.cc:729] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.009249 13971 raft_consensus.cc:482] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.009435 13861 consensus_queue.cc:260] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.009775 13971 raft_consensus.cc:504] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.010968 14007 raft_consensus.cc:482] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.011687 13971 leader_election.cc:288] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.011415 14007 raft_consensus.cc:504] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.013363 14007 leader_election.cc:288] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.015045 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.015265 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.015905 13738 raft_consensus.cc:2454] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:05.016716 13861 ts_tablet_manager.cc:1398] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20240503 02:07:05.017258 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.017477 13632 leader_election.cc:302] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:05.017869 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.018359 14007 raft_consensus.cc:2781] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.008945 13858 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.018719 14007 raft_consensus.cc:482] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.018993 13858 ts_tablet_manager.cc:1367] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.002s
I20240503 02:07:05.019070 14007 raft_consensus.cc:3037] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:05.020222 13710 leader_election.cc:341] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:05.021373 13635 leader_election.cc:341] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.021608 13858 raft_consensus.cc:348] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.021888 13957 consensus_queue.cc:1035] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.022281 13858 raft_consensus.cc:374] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.022590 13858 raft_consensus.cc:729] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.023172 13858 consensus_queue.cc:260] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.024850 13861 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.030607 13858 ts_tablet_manager.cc:1398] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:07:05.030828 14007 raft_consensus.cc:504] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.031492 13858 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.031970 13667 raft_consensus.cc:2379] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:05.035274 13709 leader_election.cc:302] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.036420 13971 raft_consensus.cc:3037] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.039445 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:05.041218 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
W20240503 02:07:05.042688 13635 leader_election.cc:341] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.044763 14007 leader_election.cc:288] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.044979 13971 raft_consensus.cc:2726] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.048820 13861 tablet_bootstrap.cc:654] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.051321 13738 raft_consensus.cc:2454] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:05.054715 13632 leader_election.cc:302] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.056838 13858 tablet_bootstrap.cc:654] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.058179 13957 raft_consensus.cc:2781] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.061682 13957 raft_consensus.cc:686] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:05.062466 13957 consensus_queue.cc:237] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.066306 13858 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.066771 13858 ts_tablet_manager.cc:1367] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20240503 02:07:05.069633 14007 consensus_queue.cc:1035] T b1ac22299784411188965b2bfbd51ef6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.074234 13860 tablet_bootstrap.cc:492] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.074909 13860 ts_tablet_manager.cc:1367] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.003s
I20240503 02:07:05.077916 13861 tablet_bootstrap.cc:492] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.077749 13860 raft_consensus.cc:348] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.078361 13861 ts_tablet_manager.cc:1367] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.000s
I20240503 02:07:05.078598 13860 raft_consensus.cc:374] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.078472 13858 raft_consensus.cc:348] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.079234 13858 raft_consensus.cc:374] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.079536 13858 raft_consensus.cc:729] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.079013 13860 raft_consensus.cc:729] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.080375 13861 raft_consensus.cc:348] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.080930 13861 raft_consensus.cc:374] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.081199 13861 raft_consensus.cc:729] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.082038 13861 consensus_queue.cc:260] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.081180 13860 consensus_queue.cc:260] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.088680 13860 ts_tablet_manager.cc:1398] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 02:07:05.080152 13858 consensus_queue.cc:260] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.090404 13858 ts_tablet_manager.cc:1398] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20240503 02:07:05.091549 13860 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.092324 13566 catalog_manager.cc:5484] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.093799 13861 ts_tablet_manager.cc:1398] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240503 02:07:05.094741 13861 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.097963 13860 tablet_bootstrap.cc:654] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.092339 13858 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.108608 13858 tablet_bootstrap.cc:654] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.121579 13861 tablet_bootstrap.cc:654] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.129593 13858 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.130017 13858 ts_tablet_manager.cc:1367] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.012s
I20240503 02:07:05.131644 13861 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.132102 13861 ts_tablet_manager.cc:1367] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.000s
I20240503 02:07:05.132530 13858 raft_consensus.cc:348] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.132879 13860 tablet_bootstrap.cc:492] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.133136 13858 raft_consensus.cc:374] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.133391 13858 raft_consensus.cc:729] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.133549 13860 ts_tablet_manager.cc:1367] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.014s
I20240503 02:07:05.133909 13858 consensus_queue.cc:260] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.134387 13861 raft_consensus.cc:348] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.135052 13861 raft_consensus.cc:374] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.135298 13861 raft_consensus.cc:729] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.135865 13858 ts_tablet_manager.cc:1398] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:05.135826 13860 raft_consensus.cc:348] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.135879 13861 consensus_queue.cc:260] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.137704 13858 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.138073 13861 ts_tablet_manager.cc:1398] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240503 02:07:05.136690 13860 raft_consensus.cc:374] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.140909 13860 raft_consensus.cc:729] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.141614 13860 consensus_queue.cc:260] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.139261 13861 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.142894 13858 tablet_bootstrap.cc:654] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.145829 13860 ts_tablet_manager.cc:1398] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240503 02:07:05.147339 13860 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.158622 13913 raft_consensus.cc:482] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.159155 13861 tablet_bootstrap.cc:654] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.159088 13913 raft_consensus.cc:504] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.162026 13913 leader_election.cc:288] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.162803 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.163583 13738 raft_consensus.cc:2379] T 4331632ebc06477a896d103b58707502 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b518e4e142f54c789a56606b766c4e06 in current term 1: Already voted for candidate 7da64b3eda0a40b6b183c1cfc19db738 in this term.
I20240503 02:07:05.165676 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4331632ebc06477a896d103b58707502" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.168445 13860 tablet_bootstrap.cc:654] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.168732 13858 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.169282 13858 ts_tablet_manager.cc:1367] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.001s
I20240503 02:07:05.171241 13783 leader_election.cc:302] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.172349 13861 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.171963 13858 raft_consensus.cc:348] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.172542 13913 raft_consensus.cc:3037] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.172705 13858 raft_consensus.cc:374] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.172832 13861 ts_tablet_manager.cc:1367] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.002s
I20240503 02:07:05.173084 13858 raft_consensus.cc:729] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.173804 13858 consensus_queue.cc:260] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.175843 13858 ts_tablet_manager.cc:1398] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:05.176930 13858 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.178999 13913 raft_consensus.cc:2726] T 4331632ebc06477a896d103b58707502 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.182236 13858 tablet_bootstrap.cc:654] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.184844 13971 raft_consensus.cc:482] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.185381 13971 raft_consensus.cc:504] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.187438 13971 leader_election.cc:288] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.189167 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.189908 13814 raft_consensus.cc:2454] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.191229 13710 leader_election.cc:302] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.193100 13971 raft_consensus.cc:2781] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.193171 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.193684 13971 raft_consensus.cc:482] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.194100 13971 raft_consensus.cc:3037] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.194183 13667 raft_consensus.cc:2454] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.206836 13971 raft_consensus.cc:504] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.209625 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.210271 13667 raft_consensus.cc:3037] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.213150 14007 raft_consensus.cc:482] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.213682 14007 raft_consensus.cc:504] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.214906 13918 raft_consensus.cc:482] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.215412 13918 raft_consensus.cc:504] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:05.222311 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080003e5a0 after lost signal to thread 12503
W20240503 02:07:05.223402 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080003ec80 after lost signal to thread 13601
W20240503 02:07:05.224697 13598 debug-util.cc:398] Leaking SignalData structure 0x7b0800255c40 after lost signal to thread 13693
W20240503 02:07:05.225914 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080014dae0 after lost signal to thread 13768
W20240503 02:07:05.226971 13598 debug-util.cc:398] Leaking SignalData structure 0x7b0800234020 after lost signal to thread 13842
I20240503 02:07:05.223338 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:05.228940 13632 leader_election.cc:341] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.229059 13918 leader_election.cc:288] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.231287 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.231798 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.231920 13814 raft_consensus.cc:2454] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.233172 13710 leader_election.cc:302] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
W20240503 02:07:05.233989 13709 leader_election.cc:341] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.233927 13861 raft_consensus.cc:348] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.234704 13861 raft_consensus.cc:374] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.234325 14007 leader_election.cc:288] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.235009 13861 raft_consensus.cc:729] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.235656 13861 consensus_queue.cc:260] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.237012 13667 raft_consensus.cc:2454] T 4f7f8c910f68477585088d11c8c0f9af P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.237707 13861 ts_tablet_manager.cc:1398] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20240503 02:07:05.238088 13709 leader_election.cc:302] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:05.238714 13921 raft_consensus.cc:2781] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.239018 13861 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
W20240503 02:07:05.227576 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080024dd60 after lost signal to thread 13999
I20240503 02:07:05.234905 13918 raft_consensus.cc:2781] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.244354 13921 raft_consensus.cc:686] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:05.246295 13918 raft_consensus.cc:482] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.247146 13918 raft_consensus.cc:3037] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.234925 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.249159 13814 raft_consensus.cc:2454] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:05.249272 13921 consensus_queue.cc:237] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.250222 13635 leader_election.cc:302] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:05.254810 13860 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.255317 13860 ts_tablet_manager.cc:1367] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.108s user 0.032s sys 0.017s
I20240503 02:07:05.251873 14007 raft_consensus.cc:2781] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.255718 14007 raft_consensus.cc:482] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.256063 14007 raft_consensus.cc:3037] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.260519 13860 raft_consensus.cc:348] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.261282 13860 raft_consensus.cc:374] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.261547 13860 raft_consensus.cc:729] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.262154 13860 consensus_queue.cc:260] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.264679 13860 ts_tablet_manager.cc:1398] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:07:05.265537 13860 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.266705 13861 tablet_bootstrap.cc:654] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.271149 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7f8c910f68477585088d11c8c0f9af" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.271965 13816 raft_consensus.cc:3037] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.272491 13971 leader_election.cc:288] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.275190 13858 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.275619 13858 ts_tablet_manager.cc:1367] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.099s user 0.022s sys 0.032s
I20240503 02:07:05.277822 13858 raft_consensus.cc:348] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.278455 13858 raft_consensus.cc:374] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.278719 13858 raft_consensus.cc:729] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.279373 13858 consensus_queue.cc:260] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.281389 13858 ts_tablet_manager.cc:1398] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:07:05.282192 13858 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.294121 13816 raft_consensus.cc:2454] T 4f7f8c910f68477585088d11c8c0f9af P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.294459 13913 raft_consensus.cc:482] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.290599 14007 raft_consensus.cc:504] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.294975 13858 tablet_bootstrap.cc:654] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.298525 13860 tablet_bootstrap.cc:654] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.299938 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:05.297958 14007 leader_election.cc:288] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.295158 13913 raft_consensus.cc:504] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:05.301291 13632 leader_election.cc:341] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.302151 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.303153 13814 raft_consensus.cc:3037] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.303351 13913 leader_election.cc:288] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.298048 13918 raft_consensus.cc:504] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.304561 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4459e1c4e013400f920bba94d281d95e" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.306408 13738 raft_consensus.cc:2454] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:05.306512 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4459e1c4e013400f920bba94d281d95e" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.308952 13667 raft_consensus.cc:2454] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:05.311100 13781 leader_election.cc:302] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.312479 14104 raft_consensus.cc:2781] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.312848 14104 raft_consensus.cc:482] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.313155 14104 raft_consensus.cc:3037] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.315690 13567 catalog_manager.cc:5484] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.323966 14104 raft_consensus.cc:504] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.325981 14104 leader_election.cc:288] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.330572 14093 raft_consensus.cc:482] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.331015 14093 raft_consensus.cc:504] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.326629 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4459e1c4e013400f920bba94d281d95e" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.332949 14093 leader_election.cc:288] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.333117 13667 raft_consensus.cc:3037] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.326890 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4459e1c4e013400f920bba94d281d95e" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:05.334059 13738 raft_consensus.cc:3037] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:05.325774 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:05.327299 13910 raft_consensus.cc:482] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.322396 13814 raft_consensus.cc:2454] T bc74b8e531c942f28a4b9467a14baf26 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:05.338918 13910 raft_consensus.cc:504] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.341078 13635 leader_election.cc:302] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:05.342190 14007 raft_consensus.cc:2781] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.342680 14007 raft_consensus.cc:686] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:05.342850 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.339308 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.344252 14007 consensus_queue.cc:237] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.344712 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:05.345398 13781 leader_election.cc:341] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.347127 13814 raft_consensus.cc:3037] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.347586 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b296978fa84ad6a6f9cc7ec6d202ac" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.348270 13861 tablet_bootstrap.cc:492] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
W20240503 02:07:05.348780 13783 leader_election.cc:341] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.349184 13861 ts_tablet_manager.cc:1367] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.009s
I20240503 02:07:05.349206 13783 leader_election.cc:302] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.342407 13910 leader_election.cc:288] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.351039 13783 leader_election.cc:302] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.355170 13861 raft_consensus.cc:348] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.352241 14093 raft_consensus.cc:2726] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.360616 13667 raft_consensus.cc:2454] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:05.359998 13861 raft_consensus.cc:374] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.361253 13861 raft_consensus.cc:729] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.361907 13861 consensus_queue.cc:260] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.363950 13861 ts_tablet_manager.cc:1398] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20240503 02:07:05.364876 13861 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.356492 13738 raft_consensus.cc:2454] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:05.365760 13783 leader_election.cc:302] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.366698 14093 raft_consensus.cc:2781] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.367336 14093 raft_consensus.cc:686] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:05.368096 14093 consensus_queue.cc:237] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.368891 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.373973 13816 raft_consensus.cc:3037] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.384965 14007 raft_consensus.cc:482] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.389516 13918 leader_election.cc:288] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.391698 13816 raft_consensus.cc:2454] T a0b623dc663b454a8380dad2c487d035 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.390269 14007 raft_consensus.cc:504] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.394027 13710 leader_election.cc:302] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.394958 14078 raft_consensus.cc:482] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.395248 14139 raft_consensus.cc:2781] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.395751 14139 raft_consensus.cc:686] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:05.395416 14078 raft_consensus.cc:504] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.396847 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.397411 14078 leader_election.cc:288] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.396525 14139 consensus_queue.cc:237] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.397882 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b623dc663b454a8380dad2c487d035" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.393944 13567 catalog_manager.cc:5484] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.398680 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.403288 13858 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.403741 13858 ts_tablet_manager.cc:1367] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.122s user 0.012s sys 0.002s
W20240503 02:07:05.405565 14140 log.cc:924] Time spent T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20240503 02:07:05.407300 13667 raft_consensus.cc:3037] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.406831 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.408406 13565 catalog_manager.cc:5484] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
W20240503 02:07:05.414625 13635 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.415393 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.416067 13741 raft_consensus.cc:2454] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
W20240503 02:07:05.416420 13632 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.417796 13632 leader_election.cc:302] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.419409 14078 raft_consensus.cc:2726] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.419420 13632 leader_election.cc:302] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:05.399958 14007 leader_election.cc:288] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.420727 13957 raft_consensus.cc:2781] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.421165 13957 raft_consensus.cc:482] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.421496 13957 raft_consensus.cc:3037] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.409471 13816 raft_consensus.cc:2454] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:05.427824 13957 raft_consensus.cc:504] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.437565 13858 raft_consensus.cc:348] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.438342 13858 raft_consensus.cc:374] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.438613 13858 raft_consensus.cc:729] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.439277 13858 consensus_queue.cc:260] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.440702 13860 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.441941 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:05.442531 13858 ts_tablet_manager.cc:1398] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.039s user 0.000s sys 0.006s
I20240503 02:07:05.447363 13858 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.442672 13957 leader_election.cc:288] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.442457 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7a151bad744d7ab3ed1effcd61e49" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.445216 13860 ts_tablet_manager.cc:1367] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.180s user 0.009s sys 0.008s
I20240503 02:07:05.448632 13741 raft_consensus.cc:3037] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.449421 13816 raft_consensus.cc:3037] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.460923 13816 raft_consensus.cc:2454] T 18a7a151bad744d7ab3ed1effcd61e49 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:05.461316 13918 raft_consensus.cc:482] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.461897 13635 leader_election.cc:302] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.461854 13918 raft_consensus.cc:504] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.463788 14078 raft_consensus.cc:2781] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.464236 14078 raft_consensus.cc:686] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:05.464998 13918 leader_election.cc:288] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.465013 14078 consensus_queue.cc:237] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.466104 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.466820 13815 raft_consensus.cc:2454] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.462049 13567 catalog_manager.cc:5484] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.454568 13667 raft_consensus.cc:2454] T a0b623dc663b454a8380dad2c487d035 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.476612 13710 leader_election.cc:302] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.477442 14139 raft_consensus.cc:2781] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.477805 14139 raft_consensus.cc:482] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.478124 14139 raft_consensus.cc:3037] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.487421 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.513779 13860 raft_consensus.cc:348] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.514731 13860 raft_consensus.cc:374] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.515012 13860 raft_consensus.cc:729] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.517220 13910 raft_consensus.cc:2726] T 67b296978fa84ad6a6f9cc7ec6d202ac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.515663 13860 consensus_queue.cc:260] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:05.507314 13709 leader_election.cc:341] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.504343 13741 raft_consensus.cc:2454] T 18a7a151bad744d7ab3ed1effcd61e49 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:05.536911 13567 catalog_manager.cc:5484] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.543401 13860 ts_tablet_manager.cc:1398] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.095s user 0.003s sys 0.004s
I20240503 02:07:05.544667 13860 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.554123 13861 tablet_bootstrap.cc:654] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.556232 13858 tablet_bootstrap.cc:654] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.569490 14139 raft_consensus.cc:504] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.585688 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
W20240503 02:07:05.587105 13709 leader_election.cc:341] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.588052 14139 leader_election.cc:288] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.589506 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.589797 13860 tablet_bootstrap.cc:654] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.590235 13814 raft_consensus.cc:3037] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.597256 14093 raft_consensus.cc:482] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.597821 14093 raft_consensus.cc:504] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.599864 14093 leader_election.cc:288] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.600479 13861 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.600991 13861 ts_tablet_manager.cc:1367] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.236s user 0.006s sys 0.007s
I20240503 02:07:05.605165 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.605837 13668 raft_consensus.cc:2454] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:05.605923 13814 raft_consensus.cc:2454] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.606921 13783 leader_election.cc:302] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.607614 14093 raft_consensus.cc:2781] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.607967 14093 raft_consensus.cc:482] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.608301 14093 raft_consensus.cc:3037] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.608835 13710 leader_election.cc:302] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.609058 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.609946 14139 raft_consensus.cc:2781] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:07:05.610213 13781 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.610478 14139 raft_consensus.cc:686] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:05.611352 14139 consensus_queue.cc:237] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.617368 14150 consensus_queue.cc:1035] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.616161 13858 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.620848 13858 ts_tablet_manager.cc:1367] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.174s user 0.014s sys 0.012s
I20240503 02:07:05.616216 14093 raft_consensus.cc:504] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.623548 13567 catalog_manager.cc:5484] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.624658 14093 leader_election.cc:288] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.625190 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.625555 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:05.625965 13668 raft_consensus.cc:3037] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.626152 13861 raft_consensus.cc:348] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:05.626509 13781 leader_election.cc:341] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.626844 13861 raft_consensus.cc:374] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.627117 13861 raft_consensus.cc:729] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.627763 13861 consensus_queue.cc:260] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.630517 13861 ts_tablet_manager.cc:1398] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20240503 02:07:05.631759 13861 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.635063 14139 raft_consensus.cc:482] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.637293 14139 raft_consensus.cc:504] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.634702 13858 raft_consensus.cc:348] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.641569 13858 raft_consensus.cc:374] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.645151 13858 raft_consensus.cc:729] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.645933 13858 consensus_queue.cc:260] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.644332 13918 raft_consensus.cc:482] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.646947 13918 raft_consensus.cc:504] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.647980 13858 ts_tablet_manager.cc:1398] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.027s user 0.004s sys 0.001s
I20240503 02:07:05.642594 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.648834 13918 leader_election.cc:288] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.649274 13858 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.651297 14150 raft_consensus.cc:482] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.651469 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.651775 14150 raft_consensus.cc:504] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.652197 13815 raft_consensus.cc:2454] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.653692 14150 leader_election.cc:288] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:05.653828 13710 leader_election.cc:341] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.654606 13710 leader_election.cc:302] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:05.655330 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.655498 13971 raft_consensus.cc:2781] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.655926 13971 raft_consensus.cc:482] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.656044 13668 raft_consensus.cc:2454] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
W20240503 02:07:05.656661 13709 leader_election.cc:341] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.656770 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.657145 13783 leader_election.cc:302] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:05.657510 13667 raft_consensus.cc:2454] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.658135 14093 raft_consensus.cc:2781] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.659096 13709 leader_election.cc:302] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.660374 14093 raft_consensus.cc:686] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:05.660662 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.661391 13918 raft_consensus.cc:2781] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:05.641392 14139 leader_election.cc:288] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.665208 13971 raft_consensus.cc:3037] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.665508 13918 raft_consensus.cc:482] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:05.669701 13918 raft_consensus.cc:3037] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.673520 13861 tablet_bootstrap.cc:654] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:05.674192 13635 leader_election.cc:341] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.677364 13860 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.677822 13860 ts_tablet_manager.cc:1367] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.133s user 0.017s sys 0.008s
I20240503 02:07:05.661415 14093 consensus_queue.cc:237] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.680001 13860 raft_consensus.cc:348] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.680675 13860 raft_consensus.cc:374] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.680976 13860 raft_consensus.cc:729] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.681543 13860 consensus_queue.cc:260] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.682294 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.683398 13860 ts_tablet_manager.cc:1398] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:07:05.684271 13860 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.685310 13971 raft_consensus.cc:504] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.689077 13971 leader_election.cc:288] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.690829 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.691108 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
W20240503 02:07:05.693670 13709 leader_election.cc:341] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.694082 13815 raft_consensus.cc:3037] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.696156 14150 consensus_queue.cc:1035] T 4331632ebc06477a896d103b58707502 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.700196 13565 catalog_manager.cc:5484] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:05.708115 13860 tablet_bootstrap.cc:654] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.708362 14150 raft_consensus.cc:482] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.711573 14150 raft_consensus.cc:504] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.713569 14150 leader_election.cc:288] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.709463 13858 tablet_bootstrap.cc:654] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.714696 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.710311 13815 raft_consensus.cc:2454] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
W20240503 02:07:05.717337 13635 leader_election.cc:341] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.716485 13710 leader_election.cc:302] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters: 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.719830 14138 raft_consensus.cc:2781] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.720310 14138 raft_consensus.cc:686] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:05.721132 14138 consensus_queue.cc:237] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.724346 13861 tablet_bootstrap.cc:492] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.724876 13861 ts_tablet_manager.cc:1367] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.003s
I20240503 02:07:05.733029 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:05.734781 13632 leader_election.cc:341] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.735198 13632 leader_election.cc:302] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.735917 14007 raft_consensus.cc:2726] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.738047 13861 raft_consensus.cc:348] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.745486 13921 consensus_queue.cc:1035] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.770324 13861 raft_consensus.cc:374] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.753779 13918 raft_consensus.cc:504] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.772486 13861 raft_consensus.cc:729] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.773298 13741 raft_consensus.cc:2379] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:05.772877 13565 catalog_manager.cc:5484] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.773747 13858 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.774363 13858 ts_tablet_manager.cc:1367] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.003s
I20240503 02:07:05.773360 13861 consensus_queue.cc:260] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.779242 13632 leader_election.cc:302] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.780504 13861 ts_tablet_manager.cc:1398] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.055s user 0.003s sys 0.002s
I20240503 02:07:05.781147 13858 raft_consensus.cc:348] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.782516 14007 raft_consensus.cc:3037] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.784373 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:05.791779 13858 raft_consensus.cc:374] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.792043 13861 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.792605 13860 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:05.793071 13860 ts_tablet_manager.cc:1367] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.000s
I20240503 02:07:05.793869 13918 leader_election.cc:288] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.795452 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:05.796077 13860 raft_consensus.cc:348] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.792613 13858 raft_consensus.cc:729] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.792796 13971 consensus_queue.cc:1035] T 4f7f8c910f68477585088d11c8c0f9af P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240503 02:07:05.799742 13910 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.796777 13860 raft_consensus.cc:374] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.801477 13860 raft_consensus.cc:729] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.797663 13858 consensus_queue.cc:260] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:05.796878 13710 leader_election.cc:341] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.814566 13910 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.817720 13861 tablet_bootstrap.cc:654] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.802134 13860 consensus_queue.cc:260] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.791532 13957 raft_consensus.cc:482] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.825670 13957 raft_consensus.cc:504] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.820923 13858 ts_tablet_manager.cc:1398] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.046s user 0.005s sys 0.002s
I20240503 02:07:05.819818 14007 raft_consensus.cc:2726] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.826463 14104 raft_consensus.cc:482] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.829363 13858 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:05.829160 14104 raft_consensus.cc:504] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.836043 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.841113 13667 raft_consensus.cc:2454] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:05.844508 14150 consensus_queue.cc:1035] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:05.846518 13709 leader_election.cc:302] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.847806 13971 raft_consensus.cc:2781] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:05.848368 13971 raft_consensus.cc:686] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:05.849303 13971 consensus_queue.cc:237] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.857187 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.857298 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:05.858441 13921 raft_consensus.cc:482] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.859129 14104 leader_election.cc:288] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:05.858992 13921 raft_consensus.cc:504] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.860163 13860 ts_tablet_manager.cc:1398] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.067s user 0.003s sys 0.003s
I20240503 02:07:05.863049 13860 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:05.860884 13921 leader_election.cc:288] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:05.858500 13783 leader_election.cc:341] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.884862 13858 tablet_bootstrap.cc:654] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:05.877590 13781 leader_election.cc:341] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:05.903664 13781 leader_election.cc:302] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:05.894996 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.905090 13662 raft_consensus.cc:2379] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:05.891624 13957 leader_election.cc:288] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:05.914479 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 55559.1us
I20240503 02:07:05.925042 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8461a9dc6344866b9e069d0b28ee64d" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.926470 13860 tablet_bootstrap.cc:654] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.931509 13861 tablet_bootstrap.cc:492] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:05.932092 13861 ts_tablet_manager.cc:1367] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.141s user 0.025s sys 0.010s
I20240503 02:07:05.921098 14093 raft_consensus.cc:2726] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:05.935775 13861 raft_consensus.cc:348] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.936547 13861 raft_consensus.cc:374] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.936877 13861 raft_consensus.cc:729] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
W20240503 02:07:05.937494 13632 consensus_peers.cc:485] T bc74b8e531c942f28a4b9467a14baf26 P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Couldn't send request to peer 65b596345545434b9f09935907294192. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:05.937757 13861 consensus_queue.cc:260] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.940279 13861 ts_tablet_manager.cc:1398] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:07:05.941141 13861 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:05.954960 13921 consensus_queue.cc:1035] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:05.958794 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74128b771f146e78210a5b8dcf823a8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:05.959589 13815 raft_consensus.cc:2379] T c74128b771f146e78210a5b8dcf823a8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:05.960232 13710 leader_election.cc:302] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.952600 13565 catalog_manager.cc:5484] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:05.963610 14162 raft_consensus.cc:3037] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.966004 13861 tablet_bootstrap.cc:654] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:05.968394 13921 consensus_queue.cc:1035] T a0b623dc663b454a8380dad2c487d035 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:05.969597 13858 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:05.971652 13858 ts_tablet_manager.cc:1367] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.142s user 0.014s sys 0.024s
I20240503 02:07:05.963073 13971 raft_consensus.cc:482] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.972132 13971 raft_consensus.cc:504] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.974989 13635 leader_election.cc:302] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:05.975788 14150 raft_consensus.cc:3037] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:05.976779 14148 consensus_queue.cc:1035] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:07:05.970132 14078 raft_consensus.cc:482] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:05.988682 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:05.989367 13668 raft_consensus.cc:2454] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:05.997272 13858 raft_consensus.cc:348] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.998001 13858 raft_consensus.cc:374] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:05.998858 13858 raft_consensus.cc:729] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:05.999949 13971 leader_election.cc:288] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:05.999352 13858 consensus_queue.cc:260] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:05.981562 14078 raft_consensus.cc:504] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.016059 13858 ts_tablet_manager.cc:1398] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.044s user 0.001s sys 0.005s
I20240503 02:07:06.017241 13858 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.020217 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.009932 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:06.024405 13710 leader_election.cc:341] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.028903 14078 leader_election.cc:288] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.029816 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d086d958d664a1ca6d0a0cc9324f480" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.030609 13816 raft_consensus.cc:2379] T 1d086d958d664a1ca6d0a0cc9324f480 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7da64b3eda0a40b6b183c1cfc19db738 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:06.030676 13709 leader_election.cc:302] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.031626 14162 raft_consensus.cc:2726] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.032078 14150 raft_consensus.cc:2726] T c74128b771f146e78210a5b8dcf823a8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.033062 13635 leader_election.cc:302] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.037963 14078 raft_consensus.cc:3037] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.033993 13860 tablet_bootstrap.cc:492] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.034749 13971 raft_consensus.cc:2781] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:06.040549 13971 raft_consensus.cc:482] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:06.040650 13860 ts_tablet_manager.cc:1367] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.178s user 0.015s sys 0.000s
I20240503 02:07:06.040975 13971 raft_consensus.cc:3037] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.052018 13860 raft_consensus.cc:348] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.053171 13860 raft_consensus.cc:374] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.053436 13860 raft_consensus.cc:729] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.054845 13860 consensus_queue.cc:260] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.062865 14078 raft_consensus.cc:2726] T 1d086d958d664a1ca6d0a0cc9324f480 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.064024 13861 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.064450 13861 ts_tablet_manager.cc:1367] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.007s
I20240503 02:07:06.065778 14104 raft_consensus.cc:482] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.066391 14104 raft_consensus.cc:504] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.073089 13861 raft_consensus.cc:348] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.073980 14104 leader_election.cc:288] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:06.075409 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.073998 13861 raft_consensus.cc:374] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.073014 13860 ts_tablet_manager.cc:1398] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20240503 02:07:06.076200 13861 raft_consensus.cc:729] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
W20240503 02:07:06.076726 13783 leader_election.cc:341] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.077003 13860 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.076880 13861 consensus_queue.cc:260] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.078851 13858 tablet_bootstrap.cc:654] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.082949 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.085240 13971 raft_consensus.cc:504] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.087294 13971 leader_election.cc:288] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.090654 13861 ts_tablet_manager.cc:1398] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240503 02:07:06.091724 14162 consensus_queue.cc:1035] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.093516 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
W20240503 02:07:06.096663 13710 leader_election.cc:341] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.096536 14093 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.098546 13861 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.102315 14148 consensus_queue.cc:1035] T 18a7a151bad744d7ab3ed1effcd61e49 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.104283 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:06.104730 13667 raft_consensus.cc:3037] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.114701 13667 raft_consensus.cc:2454] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:06.124841 13709 leader_election.cc:302] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.133512 13860 tablet_bootstrap.cc:654] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.126045 14162 raft_consensus.cc:2781] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:07:06.114595 13781 leader_election.cc:341] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.141053 14093 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.141901 13781 leader_election.cc:302] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:06.145161 14176 raft_consensus.cc:2726] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.146829 14162 raft_consensus.cc:686] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:06.159750 14162 consensus_queue.cc:237] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.175922 13971 consensus_queue.cc:1035] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.189414 14138 raft_consensus.cc:482] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.190536 14138 raft_consensus.cc:504] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.192472 14138 leader_election.cc:288] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.196416 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.197147 13818 raft_consensus.cc:2454] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:06.198527 13710 leader_election.cc:302] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:06.199370 13918 consensus_queue.cc:1035] T 1d086d958d664a1ca6d0a0cc9324f480 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.200685 13918 raft_consensus.cc:2781] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:06.201603 13918 raft_consensus.cc:482] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:06.201936 13918 raft_consensus.cc:3037] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.193301 13861 tablet_bootstrap.cc:654] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.193974 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.213765 13668 raft_consensus.cc:2454] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:06.215299 13567 catalog_manager.cc:5484] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:06.223795 13918 raft_consensus.cc:504] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.225510 14162 consensus_queue.cc:1035] T c74128b771f146e78210a5b8dcf823a8 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.231074 13858 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.231624 13858 ts_tablet_manager.cc:1367] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.215s user 0.008s sys 0.010s
I20240503 02:07:06.234830 13858 raft_consensus.cc:348] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.235540 13858 raft_consensus.cc:374] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.235806 13858 raft_consensus.cc:729] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.237875 13918 leader_election.cc:288] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.240403 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:06.241024 13667 raft_consensus.cc:3037] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.236385 13858 consensus_queue.cc:260] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.247503 13667 raft_consensus.cc:2454] T f0d6b185135347d58dfb8ddd0b758e17 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:06.248349 13858 ts_tablet_manager.cc:1398] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20240503 02:07:06.248719 13709 leader_election.cc:302] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.250349 13971 raft_consensus.cc:2781] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:06.250869 13971 raft_consensus.cc:686] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:06.251727 13971 consensus_queue.cc:237] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.276922 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d6b185135347d58dfb8ddd0b758e17" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:06.277609 13814 raft_consensus.cc:3037] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.249297 13858 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.284474 13858 tablet_bootstrap.cc:654] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.289569 13861 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.289714 14150 raft_consensus.cc:482] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.290102 13861 ts_tablet_manager.cc:1367] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.197s user 0.038s sys 0.030s
I20240503 02:07:06.290187 14150 raft_consensus.cc:504] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.292708 14150 leader_election.cc:288] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.292881 13861 raft_consensus.cc:348] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.293558 13861 raft_consensus.cc:374] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.293836 13861 raft_consensus.cc:729] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.294411 13861 consensus_queue.cc:260] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.295542 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.296144 13738 raft_consensus.cc:2454] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:06.296352 13861 ts_tablet_manager.cc:1398] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:07:06.297243 13861 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.297859 13632 leader_election.cc:302] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.298667 14150 raft_consensus.cc:2781] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:06.299018 14150 raft_consensus.cc:482] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:06.301282 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.299371 14150 raft_consensus.cc:3037] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.301594 13563 catalog_manager.cc:5484] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:06.304571 13818 raft_consensus.cc:2454] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:06.308040 14150 raft_consensus.cc:504] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.311905 13860 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.329782 13861 tablet_bootstrap.cc:654] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.330402 14078 raft_consensus.cc:482] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:07:06.333359 13710 consensus_peers.cc:485] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 -> Peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Couldn't send request to peer b518e4e142f54c789a56606b766c4e06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:06.330958 14078 raft_consensus.cc:504] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.338577 14078 leader_election.cc:288] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.338853 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:06.340461 13743 raft_consensus.cc:3037] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.345161 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:06.346699 13632 leader_election.cc:341] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.354744 13743 raft_consensus.cc:2454] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:06.315317 13860 ts_tablet_manager.cc:1367] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.238s user 0.023s sys 0.024s
I20240503 02:07:06.356091 13814 raft_consensus.cc:2454] T f0d6b185135347d58dfb8ddd0b758e17 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:06.356607 13632 leader_election.cc:302] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.361833 13957 raft_consensus.cc:2781] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:06.361912 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.362566 13957 raft_consensus.cc:686] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:06.363420 13957 consensus_queue.cc:237] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:06.367355 13635 leader_election.cc:341] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.367858 13635 leader_election.cc:302] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.368918 14078 raft_consensus.cc:2726] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.379331 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecb6a74b50e41e78a9e7d84d93b80d8" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:06.380010 13815 raft_consensus.cc:3037] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.368942 14150 leader_election.cc:288] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.392628 13860 raft_consensus.cc:348] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.393440 13860 raft_consensus.cc:374] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.393867 13860 raft_consensus.cc:729] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.397449 13815 raft_consensus.cc:2454] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:06.398145 13858 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.398639 13858 ts_tablet_manager.cc:1367] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.150s user 0.024s sys 0.011s
I20240503 02:07:06.401034 13858 raft_consensus.cc:348] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.401700 13858 raft_consensus.cc:374] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.401949 13858 raft_consensus.cc:729] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.402022 13861 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.402593 13861 ts_tablet_manager.cc:1367] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.106s user 0.022s sys 0.016s
I20240503 02:07:06.402678 13860 consensus_queue.cc:260] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.405560 13861 raft_consensus.cc:348] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.406288 13861 raft_consensus.cc:374] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.416913 13861 raft_consensus.cc:729] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.418346 13861 consensus_queue.cc:260] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.430050 13861 ts_tablet_manager.cc:1398] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20240503 02:07:06.430335 13563 catalog_manager.cc:5484] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:06.434634 13861 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.409636 13860 ts_tablet_manager.cc:1398] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
W20240503 02:07:06.437968 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:06.439833 13861 tablet_bootstrap.cc:654] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.443701 13861 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.444090 13861 ts_tablet_manager.cc:1367] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240503 02:07:06.445740 13861 raft_consensus.cc:348] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.446206 13861 raft_consensus.cc:374] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.446403 13861 raft_consensus.cc:729] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.447010 13861 consensus_queue.cc:260] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.448908 13861 ts_tablet_manager.cc:1398] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:07:06.449746 13861 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.453536 14162 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.457422 13861 tablet_bootstrap.cc:654] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.465482 13860 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.469571 14104 raft_consensus.cc:482] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.470042 14104 raft_consensus.cc:504] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.471913 14104 leader_election.cc:288] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:06.472949 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.473738 13668 raft_consensus.cc:2379] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b518e4e142f54c789a56606b766c4e06 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:06.473871 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee01df95e61d4570a6c7827cd2acc525" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.475447 13858 consensus_queue.cc:260] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.476519 13918 raft_consensus.cc:482] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.476991 13918 raft_consensus.cc:504] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.477639 13858 ts_tablet_manager.cc:1398] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20240503 02:07:06.478464 13858 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.479171 13918 leader_election.cc:288] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.480655 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:06.482074 13710 leader_election.cc:341] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.483490 13783 leader_election.cc:302] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:06.484154 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.484577 14104 raft_consensus.cc:3037] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.486703 13860 tablet_bootstrap.cc:654] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:06.490809 13709 leader_election.cc:341] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.491293 13709 leader_election.cc:302] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.492252 14162 raft_consensus.cc:2726] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.493494 14104 raft_consensus.cc:2726] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.483438 13858 tablet_bootstrap.cc:654] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.500525 13861 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.501005 13861 ts_tablet_manager.cc:1367] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.018s
I20240503 02:07:06.503789 13861 raft_consensus.cc:348] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.504427 13861 raft_consensus.cc:374] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.504678 13861 raft_consensus.cc:729] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.505336 13861 consensus_queue.cc:260] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.507524 13861 ts_tablet_manager.cc:1398] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:06.508347 13861 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.508404 13860 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.508939 13860 ts_tablet_manager.cc:1367] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.000s
I20240503 02:07:06.511209 13860 raft_consensus.cc:348] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.511879 13860 raft_consensus.cc:374] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.512130 13860 raft_consensus.cc:729] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.512692 13860 consensus_queue.cc:260] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.513859 13861 tablet_bootstrap.cc:654] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.515015 14104 raft_consensus.cc:482] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.515465 14104 raft_consensus.cc:504] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.518110 13858 tablet_bootstrap.cc:492] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.518110 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.518610 13858 ts_tablet_manager.cc:1367] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.004s
W20240503 02:07:06.519593 13783 leader_election.cc:341] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.520550 13957 raft_consensus.cc:482] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.520936 13858 raft_consensus.cc:348] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.521081 13957 raft_consensus.cc:504] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.521636 13858 raft_consensus.cc:374] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.521927 13858 raft_consensus.cc:729] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.522054 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.523511 13957 leader_election.cc:288] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:06.523449 13781 leader_election.cc:341] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.524040 13781 leader_election.cc:302] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:06.524802 14176 raft_consensus.cc:2726] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.525380 14104 leader_election.cc:288] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:06.525902 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:06.527496 13635 leader_election.cc:341] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.529178 13858 consensus_queue.cc:260] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.529457 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.529462 13860 ts_tablet_manager.cc:1398] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
W20240503 02:07:06.531775 13632 leader_election.cc:341] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.531296 13858 ts_tablet_manager.cc:1398] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240503 02:07:06.532202 13860 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.532253 13632 leader_election.cc:302] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.534361 14150 raft_consensus.cc:2726] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.537456 13860 tablet_bootstrap.cc:654] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.541102 13858 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.546097 13858 tablet_bootstrap.cc:654] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.568985 13860 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.569440 13860 ts_tablet_manager.cc:1367] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.007s
I20240503 02:07:06.569721 13861 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.570250 13861 ts_tablet_manager.cc:1367] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.020s
I20240503 02:07:06.572345 13860 raft_consensus.cc:348] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.572459 13861 raft_consensus.cc:348] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.573105 13860 raft_consensus.cc:374] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.573277 13861 raft_consensus.cc:374] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.573385 13860 raft_consensus.cc:729] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.573750 13861 raft_consensus.cc:729] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.574082 13860 consensus_queue.cc:260] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.574424 13861 consensus_queue.cc:260] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.576246 13860 ts_tablet_manager.cc:1398] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:07:06.576514 13861 ts_tablet_manager.cc:1398] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:07:06.577415 13861 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.577431 13860 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.583029 13860 tablet_bootstrap.cc:654] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.583608 13861 tablet_bootstrap.cc:654] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.570297 13858 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.593132 13858 ts_tablet_manager.cc:1367] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.001s
I20240503 02:07:06.597756 13858 raft_consensus.cc:348] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.598457 13858 raft_consensus.cc:374] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.598697 13858 raft_consensus.cc:729] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.601536 13861 tablet_bootstrap.cc:492] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.602133 13861 ts_tablet_manager.cc:1367] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.004s
I20240503 02:07:06.604606 13861 raft_consensus.cc:348] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.605451 13861 raft_consensus.cc:374] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.605772 13861 raft_consensus.cc:729] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.606545 13861 consensus_queue.cc:260] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.608489 13861 ts_tablet_manager.cc:1398] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240503 02:07:06.609658 13861 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.614401 13858 consensus_queue.cc:260] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.615640 13861 tablet_bootstrap.cc:654] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.632566 14162 raft_consensus.cc:482] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.633131 14162 raft_consensus.cc:504] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.635056 14162 leader_election.cc:288] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.637701 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.640276 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74b8e531c942f28a4b9467a14baf26" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.641700 13710 leader_election.cc:302] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.642774 14139 raft_consensus.cc:2726] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.644562 14078 raft_consensus.cc:482] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.645694 14078 raft_consensus.cc:504] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.646646 13861 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.647210 13861 ts_tablet_manager.cc:1367] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.024s
I20240503 02:07:06.649726 13861 raft_consensus.cc:348] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.650547 13861 raft_consensus.cc:374] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.650877 13861 raft_consensus.cc:729] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.651633 13861 consensus_queue.cc:260] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.654026 13861 ts_tablet_manager.cc:1398] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:07:06.655313 13861 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.660315 13861 tablet_bootstrap.cc:654] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.674017 13861 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.674547 13861 ts_tablet_manager.cc:1367] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20240503 02:07:06.676856 13861 raft_consensus.cc:348] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.677621 13861 raft_consensus.cc:374] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.678023 13861 raft_consensus.cc:729] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.678668 13861 consensus_queue.cc:260] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.682292 13858 ts_tablet_manager.cc:1398] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20240503 02:07:06.683126 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.683447 13861 ts_tablet_manager.cc:1398] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:07:06.683815 13743 raft_consensus.cc:2454] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:06.689069 13860 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.689433 13860 ts_tablet_manager.cc:1367] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.112s user 0.047s sys 0.033s
I20240503 02:07:06.691125 13860 raft_consensus.cc:348] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.691586 13860 raft_consensus.cc:374] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.691802 13860 raft_consensus.cc:729] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.692277 13860 consensus_queue.cc:260] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.693557 13860 ts_tablet_manager.cc:1398] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:07:06.694787 13632 leader_election.cc:302] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.695588 14148 raft_consensus.cc:2781] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:06.695931 14148 raft_consensus.cc:482] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:06.696251 14148 raft_consensus.cc:3037] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.700636 13860 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.703121 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.700851 14148 raft_consensus.cc:504] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.703770 13815 raft_consensus.cc:2454] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:06.705611 14148 leader_election.cc:288] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.708868 13860 tablet_bootstrap.cc:654] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.710830 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:06.711419 13814 raft_consensus.cc:3037] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.729980 13814 raft_consensus.cc:2454] T a2243e6f9b764eda92cfa44cb769fde6 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:06.730702 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2243e6f9b764eda92cfa44cb769fde6" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:06.731402 13743 raft_consensus.cc:3037] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.731874 13860 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.737067 13860 ts_tablet_manager.cc:1367] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.002s
I20240503 02:07:06.738224 14078 leader_election.cc:288] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.739926 13858 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.739926 13861 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.745534 14162 consensus_queue.cc:1035] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.739192 13957 raft_consensus.cc:482] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.748144 13861 tablet_bootstrap.cc:654] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.747884 13957 raft_consensus.cc:504] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.756263 13858 tablet_bootstrap.cc:654] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.762516 13861 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.763012 13861 ts_tablet_manager.cc:1367] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.001s
I20240503 02:07:06.766568 13861 raft_consensus.cc:348] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.767666 13957 leader_election.cc:288] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.740617 13860 raft_consensus.cc:348] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.768195 13860 raft_consensus.cc:374] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.768486 13860 raft_consensus.cc:729] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.769150 13860 consensus_queue.cc:260] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.771247 13860 ts_tablet_manager.cc:1398] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20240503 02:07:06.758800 13743 raft_consensus.cc:2454] T a2243e6f9b764eda92cfa44cb769fde6 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:06.772464 13860 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.774319 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:06.776901 13858 tablet_bootstrap.cc:492] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.777441 13858 ts_tablet_manager.cc:1367] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.004s
I20240503 02:07:06.778030 13860 tablet_bootstrap.cc:654] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.779357 13632 leader_election.cc:302] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.780149 14148 raft_consensus.cc:2781] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:06.767473 13861 raft_consensus.cc:374] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.780617 13861 raft_consensus.cc:729] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.781387 13861 consensus_queue.cc:260] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.782713 13815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.780683 14148 raft_consensus.cc:686] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
W20240503 02:07:06.786024 13632 leader_election.cc:341] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.785962 13858 raft_consensus.cc:348] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.788259 13858 raft_consensus.cc:374] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.787261 14148 consensus_queue.cc:237] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.788545 13858 raft_consensus.cc:729] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.790220 13858 consensus_queue.cc:260] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.793565 13858 ts_tablet_manager.cc:1398] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
W20240503 02:07:06.794148 13635 leader_election.cc:341] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.794652 13635 leader_election.cc:302] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.796051 13858 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.797120 14150 raft_consensus.cc:482] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.797621 14150 raft_consensus.cc:504] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.798174 13861 ts_tablet_manager.cc:1398] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20240503 02:07:06.799017 13861 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:06.799503 14150 leader_election.cc:288] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.802434 14162 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.804579 13957 raft_consensus.cc:2726] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.814529 13858 tablet_bootstrap.cc:654] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.826480 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:06.831049 13632 leader_election.cc:341] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.833108 14150 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:07:06.850409 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 56307us
I20240503 02:07:06.851352 14216 raft_consensus.cc:482] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.851900 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:06.858871 13861 tablet_bootstrap.cc:654] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:06.862903 13635 leader_election.cc:341] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.863389 13635 leader_election.cc:302] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:06.864375 13738 raft_consensus.cc:3037] T bc74b8e531c942f28a4b9467a14baf26 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.866922 13567 catalog_manager.cc:5484] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:06.873379 13860 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:06.876693 14138 consensus_queue.cc:1035] T f0d6b185135347d58dfb8ddd0b758e17 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.877278 13860 ts_tablet_manager.cc:1367] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.105s user 0.031s sys 0.007s
I20240503 02:07:06.874291 13971 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:06.867914 14078 raft_consensus.cc:2726] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:06.879572 13860 raft_consensus.cc:348] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.880237 13860 raft_consensus.cc:374] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.880496 13860 raft_consensus.cc:729] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.889065 13860 consensus_queue.cc:260] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.895359 13860 ts_tablet_manager.cc:1398] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240503 02:07:06.851852 14216 raft_consensus.cc:504] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.908216 13858 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:06.908767 13858 ts_tablet_manager.cc:1367] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.002s
I20240503 02:07:06.911343 13858 raft_consensus.cc:348] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.912042 13858 raft_consensus.cc:374] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.912302 13858 raft_consensus.cc:729] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.912998 13858 consensus_queue.cc:260] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.915798 13858 ts_tablet_manager.cc:1398] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 02:07:06.908087 14216 leader_election.cc:288] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.916815 13858 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:06.917584 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:06.918918 13632 leader_election.cc:341] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:06.919997 13860 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:06.920758 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:06.923628 13783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 73823.9us
I20240503 02:07:06.930092 13858 tablet_bootstrap.cc:654] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.926196 13861 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:06.933305 13861 ts_tablet_manager.cc:1367] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.134s user 0.019s sys 0.004s
I20240503 02:07:06.926355 14216 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 02:07:06.937331 13817 raft_consensus.cc:2454] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:06.940203 13861 raft_consensus.cc:348] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.942241 13861 raft_consensus.cc:374] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:06.942440 13861 raft_consensus.cc:729] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:06.945001 13861 consensus_queue.cc:260] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.950942 14138 raft_consensus.cc:482] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.959697 14138 raft_consensus.cc:504] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.963587 13635 leader_election.cc:302] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:06.964711 14215 raft_consensus.cc:2781] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:06.964910 14138 leader_election.cc:288] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:06.965128 14215 raft_consensus.cc:482] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:06.965464 14215 raft_consensus.cc:3037] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:06.967901 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:06.975747 13860 tablet_bootstrap.cc:654] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:06.968600 13667 raft_consensus.cc:2454] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:06.977027 14216 raft_consensus.cc:482] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:06.985122 14216 raft_consensus.cc:504] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:06.986951 13709 leader_election.cc:302] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:06.978335 13861 ts_tablet_manager.cc:1398] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.045s user 0.004s sys 0.002s
I20240503 02:07:07.009732 13861 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
W20240503 02:07:07.010444 13783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 60265.6us
I20240503 02:07:07.000180 14162 raft_consensus.cc:2781] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.008986 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:07.020498 14216 leader_election.cc:288] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:07.035097 13710 leader_election.cc:341] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.021533 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:07.036007 13818 raft_consensus.cc:2454] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:07.021770 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:07.009620 13957 raft_consensus.cc:482] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.037091 13957 raft_consensus.cc:504] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.039007 13957 leader_election.cc:288] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.045519 13635 leader_election.cc:302] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.026041 14162 raft_consensus.cc:482] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.045910 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:07.046305 14162 raft_consensus.cc:3037] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.049533 14138 raft_consensus.cc:482] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.050071 14138 raft_consensus.cc:504] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.051964 14138 leader_election.cc:288] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.052906 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac94ddbd01d4794b3711a772388f6d9" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:07.055562 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:07.057485 13710 leader_election.cc:341] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.044521 14215 raft_consensus.cc:504] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.065129 14215 leader_election.cc:288] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
W20240503 02:07:07.067675 13632 leader_election.cc:341] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:07.068507 13635 leader_election.cc:341] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.069243 13957 raft_consensus.cc:2781] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.069615 13957 raft_consensus.cc:482] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.069928 13957 raft_consensus.cc:3037] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:07.070688 13632 leader_election.cc:341] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.071132 13632 leader_election.cc:302] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738; no voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:07.072445 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:07.077729 13817 raft_consensus.cc:3037] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.078537 13957 raft_consensus.cc:504] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.080873 13957 leader_election.cc:288] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.081490 14216 raft_consensus.cc:2726] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:07.091387 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:07.092008 13818 raft_consensus.cc:3037] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.094017 13858 tablet_bootstrap.cc:492] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:07.094426 13858 ts_tablet_manager.cc:1367] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.178s user 0.026s sys 0.011s
I20240503 02:07:07.095407 13971 raft_consensus.cc:482] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.095837 13971 raft_consensus.cc:504] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.096711 13858 raft_consensus.cc:348] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.104350 13858 raft_consensus.cc:374] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:07.100317 14229 raft_consensus.cc:482] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.098639 13971 leader_election.cc:288] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.105232 14229 raft_consensus.cc:504] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.104581 13818 raft_consensus.cc:2454] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:07.106719 13635 leader_election.cc:302] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.107160 14229 leader_election.cc:288] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:07.101073 13812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:07.107553 14148 raft_consensus.cc:2781] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.107978 14148 raft_consensus.cc:686] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:07.108784 14148 consensus_queue.cc:237] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.108175 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.110884 13669 raft_consensus.cc:2454] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:07.112015 13783 leader_election.cc:302] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.114498 13861 tablet_bootstrap.cc:654] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:07.104621 13858 raft_consensus.cc:729] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
W20240503 02:07:07.116497 13632 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 45840.7us
I20240503 02:07:07.115916 13858 consensus_queue.cc:260] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.121163 14227 raft_consensus.cc:2781] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.122051 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:07.122411 14227 raft_consensus.cc:482] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.122745 14227 raft_consensus.cc:3037] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.123957 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:07.129166 13632 leader_election.cc:341] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.132201 13858 ts_tablet_manager.cc:1398] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.038s user 0.007s sys 0.001s
I20240503 02:07:07.121308 13812 raft_consensus.cc:2454] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:07.135804 13710 leader_election.cc:302] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.136943 13971 raft_consensus.cc:2781] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.137292 13971 raft_consensus.cc:482] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.137566 13971 raft_consensus.cc:3037] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.143028 13858 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192: Bootstrap starting.
W20240503 02:07:07.146587 13781 leader_election.cc:341] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.148972 13817 raft_consensus.cc:2454] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:07.150337 13635 leader_election.cc:302] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.151600 14215 raft_consensus.cc:2781] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.152036 14215 raft_consensus.cc:686] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:07.152843 14215 consensus_queue.cc:237] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.154480 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac94ddbd01d4794b3711a772388f6d9" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.154616 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.155076 13668 raft_consensus.cc:2454] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:07.101832 14162 raft_consensus.cc:504] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.131646 13743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b664b6227e0f4268a89935f722d56c8f" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:07.156343 13743 raft_consensus.cc:3037] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.156992 14162 leader_election.cc:288] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.152578 14227 raft_consensus.cc:504] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.158661 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
W20240503 02:07:07.160379 13710 leader_election.cc:341] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.160485 14227 leader_election.cc:288] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:07.162787 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:07.163173 13709 leader_election.cc:302] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:07.163345 13667 raft_consensus.cc:3037] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.145263 13971 raft_consensus.cc:504] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.166273 13971 leader_election.cc:288] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.168231 13971 raft_consensus.cc:2781] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.168527 13860 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:07.168565 13971 raft_consensus.cc:482] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.173710 13860 ts_tablet_manager.cc:1367] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.254s user 0.038s sys 0.006s
I20240503 02:07:07.173738 13971 raft_consensus.cc:3037] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.190248 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:07.190860 13818 raft_consensus.cc:3037] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.192219 13667 raft_consensus.cc:2454] T d67aeb523f5b4da696a2135fb1acb234 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:07.167266 13662 raft_consensus.cc:2454] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:07.194392 13783 leader_election.cc:302] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.195559 13667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:07.196051 13667 raft_consensus.cc:3037] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.192246 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:07.201747 14104 raft_consensus.cc:2781] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.202230 14104 raft_consensus.cc:686] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
W20240503 02:07:07.202834 13781 leader_election.cc:341] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.202994 14104 consensus_queue.cc:237] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.210492 13818 raft_consensus.cc:2454] T a321281efc6f4adf8d7fc972c85eef37 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
W20240503 02:07:07.223994 13709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 60892.4us
I20240503 02:07:07.210938 13667 raft_consensus.cc:2454] T fff18016e2e947a987c7403a319f13bc P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:07.229229 13971 raft_consensus.cc:504] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.231827 13971 leader_election.cc:288] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.232983 14176 raft_consensus.cc:482] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.235682 13709 leader_election.cc:302] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:07.237052 13971 raft_consensus.cc:2781] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.234661 14176 raft_consensus.cc:504] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.238808 13971 raft_consensus.cc:686] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:07.239529 13971 consensus_queue.cc:237] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.241811 13858 tablet_bootstrap.cc:654] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:07.239650 13567 catalog_manager.cc:5484] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:07.233867 13710 leader_election.cc:302] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:07.252259 13743 raft_consensus.cc:2454] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
W20240503 02:07:07.242900 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 57400.2us
I20240503 02:07:07.256379 14176 leader_election.cc:288] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:07.260815 13860 raft_consensus.cc:348] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.263476 13860 raft_consensus.cc:374] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:07.263764 13860 raft_consensus.cc:729] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:07.256443 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.267917 13860 consensus_queue.cc:260] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.264731 14138 raft_consensus.cc:2781] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.300158 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a321281efc6f4adf8d7fc972c85eef37" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:07.305394 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac94ddbd01d4794b3711a772388f6d9" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:07.305999 13668 raft_consensus.cc:3037] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.309103 13860 ts_tablet_manager.cc:1398] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.135s user 0.006s sys 0.000s
I20240503 02:07:07.305400 13662 raft_consensus.cc:3037] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.282804 14138 raft_consensus.cc:686] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:07.365969 13860 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:07.363974 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97598aefb41c4115a37385063f8d0eb7" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:07.379213 13668 raft_consensus.cc:2454] T 6ac94ddbd01d4794b3711a772388f6d9 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
W20240503 02:07:07.380450 13709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 72820.3us
I20240503 02:07:07.385372 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac94ddbd01d4794b3711a772388f6d9" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:07.386891 13709 leader_election.cc:302] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:07.388612 13918 raft_consensus.cc:2781] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:07.387575 13662 raft_consensus.cc:2454] T a321281efc6f4adf8d7fc972c85eef37 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:07.434705 13565 catalog_manager.cc:5484] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:07.396653 13957 consensus_queue.cc:1035] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20240503 02:07:07.453595 13918 raft_consensus.cc:686] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:07.455653 13818 raft_consensus.cc:3037] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.456679 13858 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:07.457165 13858 ts_tablet_manager.cc:1367] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.314s user 0.035s sys 0.012s
I20240503 02:07:07.454421 13918 consensus_queue.cc:237] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:07.457813 14236 log.cc:924] Time spent T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20240503 02:07:07.470366 13860 tablet_bootstrap.cc:654] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:07.433065 14138 consensus_queue.cc:237] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.494653 13858 raft_consensus.cc:348] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.495497 13858 raft_consensus.cc:374] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:07.495759 13858 raft_consensus.cc:729] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
W20240503 02:07:07.499488 13710 leader_election.cc:341] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.496351 13858 consensus_queue.cc:260] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.503033 13858 ts_tablet_manager.cc:1398] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20240503 02:07:07.554188 14229 raft_consensus.cc:482] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 02:07:07.614826 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 45099.1us
I20240503 02:07:07.622642 13781 leader_election.cc:302] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
W20240503 02:07:07.560241 13710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 54376.3us
I20240503 02:07:07.637776 14229 raft_consensus.cc:504] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.672943 14249 raft_consensus.cc:482] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:07.673161 14229 leader_election.cc:288] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:07.682153 13858 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192: Bootstrap starting.
W20240503 02:07:07.686643 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 225915us
I20240503 02:07:07.746140 14253 raft_consensus.cc:2726] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:07.708295 14249 raft_consensus.cc:504] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.735868 14258 consensus_queue.cc:1035] T a2243e6f9b764eda92cfa44cb769fde6 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240503 02:07:07.776046 13858 tablet_bootstrap.cc:654] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
W20240503 02:07:07.789291 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 88664us
I20240503 02:07:07.793248 13861 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:07.793717 13861 ts_tablet_manager.cc:1367] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.784s user 0.026s sys 0.022s
I20240503 02:07:07.796836 13860 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:07.797613 13860 ts_tablet_manager.cc:1367] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.432s user 0.031s sys 0.027s
I20240503 02:07:07.798972 13861 raft_consensus.cc:348] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.799726 13861 raft_consensus.cc:374] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:07.799985 13861 raft_consensus.cc:729] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:07.796305 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:07.800593 13861 consensus_queue.cc:260] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:07.835085 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 50636.9us
I20240503 02:07:07.841717 14248 consensus_queue.cc:1035] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
W20240503 02:07:07.844923 13781 leader_election.cc:341] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:07.858186 13861 ts_tablet_manager.cc:1398] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.064s user 0.005s sys 0.001s
I20240503 02:07:07.862442 13860 raft_consensus.cc:348] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.863243 13860 raft_consensus.cc:374] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:07.863524 13860 raft_consensus.cc:729] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:07.864185 13860 consensus_queue.cc:260] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:07.884913 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.885588 13668 raft_consensus.cc:2454] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:07.886770 13860 ts_tablet_manager.cc:1398] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
W20240503 02:07:07.891424 13632 consensus_peers.cc:485] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Couldn't send request to peer 65b596345545434b9f09935907294192. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:07.906152 14249 leader_election.cc:288] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:07.918592 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:07.919615 13662 raft_consensus.cc:2454] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:07.866719 14248 consensus_queue.cc:1035] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240503 02:07:07.927933 13783 leader_election.cc:302] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:07.941341 14259 raft_consensus.cc:2781] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.941877 14259 raft_consensus.cc:482] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.942209 14259 raft_consensus.cc:3037] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:07.945325 13709 leader_election.cc:302] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:07.951483 13918 raft_consensus.cc:2781] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:07.951905 13918 raft_consensus.cc:482] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:07.952111 13918 raft_consensus.cc:3037] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:07.958129 13615 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:07:07.961954 13860 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
W20240503 02:07:07.949887 13710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 106776us
I20240503 02:07:07.949488 14235 consensus_queue.cc:1035] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240503 02:07:07.985108 13918 raft_consensus.cc:504] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:07.995898 13781 consensus_peers.cc:485] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 -> Peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Couldn't send request to peer 65b596345545434b9f09935907294192. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:08.000805 13918 leader_election.cc:288] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.020793 13858 tablet_bootstrap.cc:492] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.021412 13858 ts_tablet_manager.cc:1367] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.339s user 0.016s sys 0.014s
I20240503 02:07:08.036235 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:08.036963 13661 raft_consensus.cc:3037] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.050827 14229 raft_consensus.cc:482] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.050048 14249 consensus_queue.cc:1035] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240503 02:07:08.068343 14229 raft_consensus.cc:504] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.069620 13860 tablet_bootstrap.cc:654] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.070345 14229 leader_election.cc:288] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:08.047572 14259 raft_consensus.cc:504] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:08.105274 13710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 62011.1us
I20240503 02:07:08.132891 14259 leader_election.cc:288] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:08.143137 13858 raft_consensus.cc:348] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.145421 13858 raft_consensus.cc:374] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.145797 13858 raft_consensus.cc:729] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.146499 13858 consensus_queue.cc:260] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:08.156386 13710 consensus_peers.cc:485] T fff18016e2e947a987c7403a319f13bc P 65b596345545434b9f09935907294192 -> Peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Couldn't send request to peer b518e4e142f54c789a56606b766c4e06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:08.164059 13858 ts_tablet_manager.cc:1398] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.142s user 0.003s sys 0.005s
I20240503 02:07:08.164397 14247 consensus_queue.cc:1035] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240503 02:07:08.164355 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:08.167699 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feefafcc54784de88a55414462073851" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:08.182871 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
W20240503 02:07:08.188032 13781 leader_election.cc:341] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.188671 13566 catalog_manager.cc:5484] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.190865 13661 raft_consensus.cc:2454] T feefafcc54784de88a55414462073851 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:08.191907 13709 leader_election.cc:302] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters:
I20240503 02:07:08.196285 14235 consensus_queue.cc:1035] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:07:08.198684 14280 raft_consensus.cc:2781] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:08.199843 14280 raft_consensus.cc:686] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:08.200664 14280 consensus_queue.cc:237] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:08.199419 13710 leader_election.cc:341] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:08.224519 13710 leader_election.cc:341] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:07:08.225514 13710 consensus_peers.cc:485] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 -> Peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Couldn't send request to peer b518e4e142f54c789a56606b766c4e06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:08.225667 13861 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738: Bootstrap starting.
I20240503 02:07:08.242209 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:08.245460 13668 raft_consensus.cc:3037] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.256868 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:08.260306 13781 leader_election.cc:341] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.273406 13861 tablet_bootstrap.cc:654] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.283437 13860 tablet_bootstrap.cc:492] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:08.283394 14266 consensus_queue.cc:1035] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20240503 02:07:08.284142 13860 ts_tablet_manager.cc:1367] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.322s user 0.029s sys 0.017s
I20240503 02:07:08.295302 13860 raft_consensus.cc:348] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.296083 13860 raft_consensus.cc:374] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.296337 13860 raft_consensus.cc:729] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.296942 13860 consensus_queue.cc:260] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.311995 13860 ts_tablet_manager.cc:1398] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.028s user 0.002s sys 0.003s
I20240503 02:07:08.312906 13860 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:08.340991 13668 raft_consensus.cc:2454] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:08.393714 14269 consensus_queue.cc:1035] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240503 02:07:08.399523 13861 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738: No bootstrap required, opened a new log
I20240503 02:07:08.400130 13861 ts_tablet_manager.cc:1367] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent bootstrapping tablet: real 0.175s user 0.009s sys 0.007s
I20240503 02:07:08.402418 13860 tablet_bootstrap.cc:654] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.404651 13663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.404939 13858 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:08.412489 13861 raft_consensus.cc:348] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.413201 13861 raft_consensus.cc:374] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.413475 13861 raft_consensus.cc:729] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.414106 13861 consensus_queue.cc:260] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.415854 13861 ts_tablet_manager.cc:1398] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240503 02:07:08.421595 13858 tablet_bootstrap.cc:654] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.439680 14249 raft_consensus.cc:482] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.443236 14249 raft_consensus.cc:504] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.445348 14249 leader_election.cc:288] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.447924 13663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.448160 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c6c7248bdc40acad0bd1dd3ad6cb78" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:08.450088 13710 leader_election.cc:302] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:08.451231 14249 raft_consensus.cc:2726] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:08.458977 13860 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:08.459458 13860 ts_tablet_manager.cc:1367] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.147s user 0.017s sys 0.003s
I20240503 02:07:08.461725 13860 raft_consensus.cc:348] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.462390 13860 raft_consensus.cc:374] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.462625 13860 raft_consensus.cc:729] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.463188 13860 consensus_queue.cc:260] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.466233 13565 catalog_manager.cc:5484] T d67aeb523f5b4da696a2135fb1acb234 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:08.470060 13860 ts_tablet_manager.cc:1398] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240503 02:07:08.473407 13860 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:08.478886 13860 tablet_bootstrap.cc:654] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.483973 13783 leader_election.cc:302] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:08.485028 14239 raft_consensus.cc:2781] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:08.485496 14239 raft_consensus.cc:686] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:08.486243 14239 consensus_queue.cc:237] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.490166 13858 tablet_bootstrap.cc:492] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.490641 13858 ts_tablet_manager.cc:1367] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.001s
I20240503 02:07:08.493009 13858 raft_consensus.cc:348] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.493695 13858 raft_consensus.cc:374] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.493965 13858 raft_consensus.cc:729] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.494630 13858 consensus_queue.cc:260] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.496917 13858 ts_tablet_manager.cc:1398] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240503 02:07:08.503144 13858 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192: Bootstrap starting.
W20240503 02:07:08.498992 13783 leader_election.cc:341] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.505146 13783 leader_election.cc:302] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:08.506100 14220 raft_consensus.cc:2726] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:08.508226 13858 tablet_bootstrap.cc:654] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.523353 14216 raft_consensus.cc:482] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.523866 14216 raft_consensus.cc:504] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.526881 14216 leader_election.cc:288] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.528637 13735 raft_consensus.cc:3037] T 95c6c7248bdc40acad0bd1dd3ad6cb78 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.533582 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:08.534207 13818 raft_consensus.cc:2454] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 0.
I20240503 02:07:08.535187 13635 leader_election.cc:302] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:08.535895 14216 raft_consensus.cc:2781] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:08.536458 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:08.536998 14216 raft_consensus.cc:482] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 02:07:08.538540 13632 leader_election.cc:341] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.538110 14216 raft_consensus.cc:3037] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.539909 13858 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.541898 13858 ts_tablet_manager.cc:1367] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.002s
I20240503 02:07:08.546919 14216 raft_consensus.cc:504] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.556839 13858 raft_consensus.cc:348] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.557636 13858 raft_consensus.cc:374] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.557886 13858 raft_consensus.cc:729] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.558485 13858 consensus_queue.cc:260] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.559108 13818 raft_consensus.cc:3037] T 6ac94ddbd01d4794b3711a772388f6d9 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.563370 13858 ts_tablet_manager.cc:1398] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20240503 02:07:08.565766 13567 catalog_manager.cc:5484] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.567922 13858 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:08.567303 13567 catalog_manager.cc:5484] T a321281efc6f4adf8d7fc972c85eef37 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.569181 13567 catalog_manager.cc:5484] T 6ac94ddbd01d4794b3711a772388f6d9 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.573519 13860 tablet_bootstrap.cc:492] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:08.574009 13860 ts_tablet_manager.cc:1367] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.101s user 0.017s sys 0.003s
I20240503 02:07:08.575837 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
I20240503 02:07:08.579248 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e9257172234c59a0ad9b65af11d512" candidate_uuid: "7da64b3eda0a40b6b183c1cfc19db738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:08.579843 13811 raft_consensus.cc:3037] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.579079 13860 raft_consensus.cc:348] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
W20240503 02:07:08.581063 13632 leader_election.cc:341] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.581403 13860 raft_consensus.cc:374] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.577203 14216 leader_election.cc:288] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Requested vote from peers 65b596345545434b9f09935907294192 (127.12.53.2:42457), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.592072 13860 raft_consensus.cc:729] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.594135 13860 consensus_queue.cc:260] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.596171 13860 ts_tablet_manager.cc:1398] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20240503 02:07:08.597115 13860 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:08.597677 13811 raft_consensus.cc:2454] T 96e9257172234c59a0ad9b65af11d512 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da64b3eda0a40b6b183c1cfc19db738 in term 1.
I20240503 02:07:08.598810 13635 leader_election.cc:302] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:08.599828 14267 raft_consensus.cc:2781] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:08.600279 14267 raft_consensus.cc:686] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Becoming Leader. State: Replica: 7da64b3eda0a40b6b183c1cfc19db738, State: Running, Role: LEADER
I20240503 02:07:08.601089 14267 consensus_queue.cc:237] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.608291 14259 raft_consensus.cc:482] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.608821 14259 raft_consensus.cc:504] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.610924 14259 leader_election.cc:288] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:08.611960 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
W20240503 02:07:08.613086 13781 leader_election.cc:341] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.614167 13660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.614871 13660 raft_consensus.cc:2454] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 0.
I20240503 02:07:08.615971 13783 leader_election.cc:302] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192
I20240503 02:07:08.616991 14259 raft_consensus.cc:2781] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:08.617374 14259 raft_consensus.cc:482] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:08.617707 14259 raft_consensus.cc:3037] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.628032 14259 raft_consensus.cc:504] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.629009 13918 raft_consensus.cc:482] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.629446 13918 raft_consensus.cc:504] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.630684 13660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:08.631103 13918 leader_election.cc:288] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.631260 13660 raft_consensus.cc:3037] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.635156 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
W20240503 02:07:08.636588 13710 leader_election.cc:341] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.639429 13660 raft_consensus.cc:2454] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b518e4e142f54c789a56606b766c4e06 in term 1.
I20240503 02:07:08.641862 13783 leader_election.cc:302] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06; no voters:
I20240503 02:07:08.642624 14093 raft_consensus.cc:2781] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:08.643096 14093 raft_consensus.cc:686] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Becoming Leader. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Running, Role: LEADER
I20240503 02:07:08.643895 14093 consensus_queue.cc:237] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.627929 13858 tablet_bootstrap.cc:654] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.647924 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.648651 13662 raft_consensus.cc:2454] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 0.
I20240503 02:07:08.649447 14259 leader_election.cc:288] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:08.652864 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2365517fa2742fead22e9bceb13af2d" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192"
W20240503 02:07:08.656442 13781 leader_election.cc:341] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65b596345545434b9f09935907294192 (127.12.53.2:42457): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.655812 13565 catalog_manager.cc:5484] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:08.659446 13709 leader_election.cc:302] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:08.662398 14249 raft_consensus.cc:2781] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:07:08.662751 14249 raft_consensus.cc:482] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:07:08.663085 14249 raft_consensus.cc:3037] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.668314 13567 catalog_manager.cc:5484] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1). New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.677790 14249 raft_consensus.cc:504] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.680305 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738"
I20240503 02:07:08.680593 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06"
I20240503 02:07:08.680927 13662 raft_consensus.cc:3037] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:07:08.681901 13710 leader_election.cc:341] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:07:08.688539 13918 consensus_queue.cc:1035] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:08.696141 13860 tablet_bootstrap.cc:654] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.692919 13567 catalog_manager.cc:5484] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 reported cstate change: term changed from 0 to 1, leader changed from <none> to b518e4e142f54c789a56606b766c4e06 (127.12.53.3). New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } }
I20240503 02:07:08.701432 14249 leader_election.cc:288] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Requested vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.722131 13662 raft_consensus.cc:2454] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b596345545434b9f09935907294192 in term 1.
I20240503 02:07:08.724363 13709 leader_election.cc:302] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738; no voters: b518e4e142f54c789a56606b766c4e06
I20240503 02:07:08.724802 13858 tablet_bootstrap.cc:492] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.725036 14249 raft_consensus.cc:2781] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:07:08.725271 13858 ts_tablet_manager.cc:1367] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.158s user 0.011s sys 0.001s
I20240503 02:07:08.727003 14249 raft_consensus.cc:686] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Becoming Leader. State: Replica: 65b596345545434b9f09935907294192, State: Running, Role: LEADER
I20240503 02:07:08.727902 14249 consensus_queue.cc:237] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.733698 13858 raft_consensus.cc:348] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.734433 13858 raft_consensus.cc:374] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.734709 13858 raft_consensus.cc:729] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.735321 13858 consensus_queue.cc:260] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.742141 13816 raft_consensus.cc:3037] T feefafcc54784de88a55414462073851 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.745658 13860 tablet_bootstrap.cc:492] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:08.746196 13860 ts_tablet_manager.cc:1367] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.149s user 0.008s sys 0.011s
I20240503 02:07:08.747176 13858 ts_tablet_manager.cc:1398] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20240503 02:07:08.748613 13860 raft_consensus.cc:348] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.749415 13860 raft_consensus.cc:374] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.749680 13860 raft_consensus.cc:729] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.753940 13858 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:08.766104 13860 consensus_queue.cc:260] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.772626 13858 tablet_bootstrap.cc:654] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.774224 14249 consensus_queue.cc:1035] T feefafcc54784de88a55414462073851 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:08.780692 13566 catalog_manager.cc:5484] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b596345545434b9f09935907294192 (127.12.53.2). New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:08.782155 13860 ts_tablet_manager.cc:1398] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20240503 02:07:08.782831 13860 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06: Bootstrap starting.
I20240503 02:07:08.805910 14259 raft_consensus.cc:482] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.806493 14259 raft_consensus.cc:504] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.808480 14259 leader_election.cc:288] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:08.810148 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:08.811697 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff18016e2e947a987c7403a319f13bc" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.815078 13858 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.814920 13781 leader_election.cc:302] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:08.816088 13860 tablet_bootstrap.cc:654] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.826431 13816 raft_consensus.cc:3037] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.826789 13858 ts_tablet_manager.cc:1367] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.009s
I20240503 02:07:08.829128 13858 raft_consensus.cc:348] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.829788 14249 raft_consensus.cc:482] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:08.837116 14249 raft_consensus.cc:504] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.840617 13660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:08.841652 13660 raft_consensus.cc:2379] T af476a0cb51d478b952e5c4fb44447e2 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65b596345545434b9f09935907294192 in current term 1: Already voted for candidate b518e4e142f54c789a56606b766c4e06 in this term.
I20240503 02:07:08.843523 14249 leader_election.cc:288] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:08.844166 13858 raft_consensus.cc:374] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.844497 13858 raft_consensus.cc:729] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.845189 13858 consensus_queue.cc:260] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.845649 13818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af476a0cb51d478b952e5c4fb44447e2" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:08.847399 13858 ts_tablet_manager.cc:1398] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240503 02:07:08.847944 13710 leader_election.cc:302] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:08.848955 14249 raft_consensus.cc:3037] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.856372 13860 tablet_bootstrap.cc:492] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06: No bootstrap required, opened a new log
I20240503 02:07:08.858184 13860 ts_tablet_manager.cc:1367] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.006s
I20240503 02:07:08.854116 14249 raft_consensus.cc:2726] T af476a0cb51d478b952e5c4fb44447e2 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:08.857085 13858 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192: Bootstrap starting.
I20240503 02:07:08.860486 13735 raft_consensus.cc:3037] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:08.861346 13860 raft_consensus.cc:348] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.862098 13860 raft_consensus.cc:374] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.862403 13860 raft_consensus.cc:729] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b518e4e142f54c789a56606b766c4e06, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.863145 13860 consensus_queue.cc:260] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.866168 13860 ts_tablet_manager.cc:1398] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:07:08.889055 14259 raft_consensus.cc:2726] T fff18016e2e947a987c7403a319f13bc P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:08.893302 13858 tablet_bootstrap.cc:654] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192: Neither blocks nor log segments found. Creating new log.
I20240503 02:07:08.897884 14266 consensus_queue.cc:1035] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:07:08.942764 13858 tablet_bootstrap.cc:492] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192: No bootstrap required, opened a new log
I20240503 02:07:08.943222 13858 ts_tablet_manager.cc:1367] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.000s
I20240503 02:07:08.945670 13858 raft_consensus.cc:348] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.946368 13858 raft_consensus.cc:374] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:07:08.946624 13858 raft_consensus.cc:729] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b596345545434b9f09935907294192, State: Initialized, Role: FOLLOWER
I20240503 02:07:08.947203 13858 consensus_queue.cc:260] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.949623 13858 ts_tablet_manager.cc:1398] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:07:08.949366 14311 consensus_queue.cc:1035] T af476a0cb51d478b952e5c4fb44447e2 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:08.992544 14249 raft_consensus.cc:482] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b518e4e142f54c789a56606b766c4e06)
I20240503 02:07:08.993090 14249 raft_consensus.cc:504] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:08.999099 14249 leader_election.cc:288] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:09.000859 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b518e4e142f54c789a56606b766c4e06" is_pre_election: true
I20240503 02:07:09.003602 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67aeb523f5b4da696a2135fb1acb234" candidate_uuid: "65b596345545434b9f09935907294192" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:09.004858 13709 leader_election.cc:302] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65b596345545434b9f09935907294192; no voters: 7da64b3eda0a40b6b183c1cfc19db738, b518e4e142f54c789a56606b766c4e06
I20240503 02:07:09.007862 14280 raft_consensus.cc:2726] T d67aeb523f5b4da696a2135fb1acb234 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:07:09.018884 13735 raft_consensus.cc:3037] T 96e9257172234c59a0ad9b65af11d512 P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:09.028772 14267 consensus_queue.cc:1035] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.040994 14267 consensus_queue.cc:1035] T 96e9257172234c59a0ad9b65af11d512 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.081357 14311 raft_consensus.cc:482] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:07:09.081843 14311 raft_consensus.cc:504] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } }
I20240503 02:07:09.083954 14311 leader_election.cc:288] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671), 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:09.084368 13669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da64b3eda0a40b6b183c1cfc19db738" is_pre_election: true
I20240503 02:07:09.084527 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e254fe56b844e41b2abbf3626923726" candidate_uuid: "b518e4e142f54c789a56606b766c4e06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65b596345545434b9f09935907294192" is_pre_election: true
I20240503 02:07:09.085283 13669 raft_consensus.cc:2379] T 7e254fe56b844e41b2abbf3626923726 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b518e4e142f54c789a56606b766c4e06 in current term 1: Already voted for candidate 65b596345545434b9f09935907294192 in this term.
I20240503 02:07:09.086192 13783 leader_election.cc:302] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b518e4e142f54c789a56606b766c4e06; no voters: 65b596345545434b9f09935907294192, 7da64b3eda0a40b6b183c1cfc19db738
I20240503 02:07:09.086882 14311 raft_consensus.cc:3037] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:09.091667 14311 raft_consensus.cc:2726] T 7e254fe56b844e41b2abbf3626923726 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:07:09.099783 14311 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.111850 13735 raft_consensus.cc:3037] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:07:09.122949 14266 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.151099 14249 consensus_queue.cc:1035] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.162191 14162 consensus_queue.cc:1035] T 7e254fe56b844e41b2abbf3626923726 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:07:09.946168 12500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:07:09.981940 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:07:09.984328 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:07:09.985255 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:07:09.986053 12500 server_base.cc:1034] running on GCE node
I20240503 02:07:09.987866 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:07:09.988190 12500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:07:09.988358 12500 hybrid_clock.cc:648] HybridClock initialized: now 1714702029988340 us; error 0 us; skew 500 ppm
I20240503 02:07:09.988999 12500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:07:10.002640 12500 webserver.cc:460] Webserver started at http://127.12.53.4:39539/ using document root <none> and password file <none>
I20240503 02:07:10.003161 12500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:07:10.004161 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:07:10.004632 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:07:10.005831 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-3-root/instance:
uuid: "bd5512acd2bf412db3c2fbcffbf3b111"
format_stamp: "Formatted at 2024-05-03 02:07:10 on dist-test-slave-fmt4"
I20240503 02:07:10.013832 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240503 02:07:10.062502 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:07:10.064653 12500 fs_manager.cc:725] Time spent opening block manager: real 0.048s user 0.001s sys 0.010s
I20240503 02:07:10.065868 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-3-root
uuid: "bd5512acd2bf412db3c2fbcffbf3b111"
format_stamp: "Formatted at 2024-05-03 02:07:10 on dist-test-slave-fmt4"
I20240503 02:07:10.067438 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:07:10.237655 12500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:07:10.242540 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:07:10.257982 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:07:10.286623 12500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:07:10.286954 12500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:07:10.289037 12500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:07:10.289314 12500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:07:10.445437 12500 rpc_server.cc:307] RPC server started. Bound to: 127.12.53.4:41693
I20240503 02:07:10.595757 14399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.53.4:41693 every 8 connection(s)
I20240503 02:07:10.669606 12500 auto_leader_rebalancer-test.cc:405] add a tserver: bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:10.685523 14400 heartbeater.cc:347] Connected to a master server at 127.12.53.62:37853
I20240503 02:07:10.685935 14400 heartbeater.cc:464] Registering TS with master...
I20240503 02:07:10.686677 14400 heartbeater.cc:511] Master 127.12.53.62:37853 requested a full tablet report, sending...
I20240503 02:07:10.688603 13566 ts_manager.cc:194] Registered new tserver with Master: bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693)
I20240503 02:07:10.690310 13566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59688
I20240503 02:07:11.427851 13660 consensus_queue.cc:237] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:11.437194 13816 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.439133 14341 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:07:11.441066 13737 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.444837 14341 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:11.454941 14341 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.462954 13816 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
W20240503 02:07:11.479342 13635 consensus_peers.cc:485] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ecb6a74b50e41e78a9e7d84d93b80d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:11.486776 13737 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.483879 13566 catalog_manager.cc:5484] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.503598 13741 consensus_queue.cc:237] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:11.518205 13664 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.518682 13812 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.521607 14162 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:07:11.523723 14162 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:07:11.543428 13710 consensus_peers.cc:485] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee01df95e61d4570a6c7827cd2acc525. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:11.545164 14410 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.549538 13812 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.559428 13741 consensus_queue.cc:237] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:11.559659 13664 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.577106 13565 catalog_manager.cc:5484] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:11.599723 13664 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:11.601100 13710 consensus_peers.cc:485] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fed43100e614122b8bbc0bc3506d664. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:11.601658 14410 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:11.604897 13812 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.606267 14410 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:11.629374 14409 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.630959 13660 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.635833 13816 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.656852 13565 catalog_manager.cc:5484] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.668154 13816 consensus_queue.cc:237] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:11.683789 13737 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.685684 14332 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:07:11.693109 14400 heartbeater.cc:503] Master 127.12.53.62:37853 was elected leader, sending a full tablet report...
I20240503 02:07:11.700487 13660 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:11.703779 14332 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:11.709991 14093 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
W20240503 02:07:11.716207 13784 consensus_peers.cc:485] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8461a9dc6344866b9e069d0b28ee64d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:11.720333 13737 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:11.737756 13567 catalog_manager.cc:5484] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:11.759912 13664 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:12.064051 14438 ts_tablet_manager.cc:902] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671)
I20240503 02:07:12.073861 14439 ts_tablet_manager.cc:902] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:12.115923 14438 tablet_copy_client.cc:308] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.1:33671
I20240503 02:07:12.138607 14444 ts_tablet_manager.cc:902] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:12.154945 13679 tablet_copy_service.cc:140] P 7da64b3eda0a40b6b183c1cfc19db738: Received BeginTabletCopySession request for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:55890)
I20240503 02:07:12.155663 13679 tablet_copy_service.cc:161] P 7da64b3eda0a40b6b183c1cfc19db738: Beginning new tablet copy session on tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:55890: session id = bd5512acd2bf412db3c2fbcffbf3b111-1ecb6a74b50e41e78a9e7d84d93b80d8
I20240503 02:07:12.177793 13679 tablet_copy_source_session.cc:205] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:12.181785 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:12.198276 14438 tablet_copy_client.cc:783] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:12.198945 14438 tablet_copy_client.cc:647] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:12.203055 14438 tablet_copy_client.cc:523] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:12.203452 14444 tablet_copy_client.cc:308] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.2:42457
I20240503 02:07:12.206907 14439 tablet_copy_client.cc:308] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.2:42457
I20240503 02:07:12.211881 14438 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:12.233939 13754 tablet_copy_service.cc:140] P 65b596345545434b9f09935907294192: Received BeginTabletCopySession request for tablet 2fed43100e614122b8bbc0bc3506d664 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:44274)
I20240503 02:07:12.234341 13753 tablet_copy_service.cc:140] P 65b596345545434b9f09935907294192: Received BeginTabletCopySession request for tablet ee01df95e61d4570a6c7827cd2acc525 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:44274)
I20240503 02:07:12.234685 13754 tablet_copy_service.cc:161] P 65b596345545434b9f09935907294192: Beginning new tablet copy session on tablet 2fed43100e614122b8bbc0bc3506d664 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:44274: session id = bd5512acd2bf412db3c2fbcffbf3b111-2fed43100e614122b8bbc0bc3506d664
I20240503 02:07:12.239775 13753 tablet_copy_service.cc:161] P 65b596345545434b9f09935907294192: Beginning new tablet copy session on tablet ee01df95e61d4570a6c7827cd2acc525 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:44274: session id = bd5512acd2bf412db3c2fbcffbf3b111-ee01df95e61d4570a6c7827cd2acc525
I20240503 02:07:12.250098 13753 tablet_copy_source_session.cc:205] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:12.253106 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee01df95e61d4570a6c7827cd2acc525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:12.259248 14438 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:12.260264 14438 tablet_bootstrap.cc:492] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:12.260967 14438 ts_tablet_manager.cc:1367] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.049s user 0.020s sys 0.018s
I20240503 02:07:12.263846 14438 raft_consensus.cc:348] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.264690 14438 raft_consensus.cc:729] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:12.265353 14438 consensus_queue.cc:260] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.266153 13754 tablet_copy_source_session.cc:205] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:12.269178 14444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed43100e614122b8bbc0bc3506d664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:12.271627 14439 tablet_copy_client.cc:783] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:12.272197 14439 tablet_copy_client.cc:647] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:12.275735 14439 tablet_copy_client.cc:523] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:12.288048 14439 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:12.289803 14444 tablet_copy_client.cc:783] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:12.290414 14444 tablet_copy_client.cc:647] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:12.296105 14444 tablet_copy_client.cc:523] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:12.306475 14438 ts_tablet_manager.cc:1398] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.045s user 0.017s sys 0.009s
I20240503 02:07:12.311269 14444 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:12.314208 13679 tablet_copy_service.cc:342] P 7da64b3eda0a40b6b183c1cfc19db738: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-1ecb6a74b50e41e78a9e7d84d93b80d8 received from {username='slave'} at 127.0.0.1:55890
I20240503 02:07:12.314754 13679 tablet_copy_service.cc:433] P 7da64b3eda0a40b6b183c1cfc19db738: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-1ecb6a74b50e41e78a9e7d84d93b80d8 on tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:12.339630 14438 ts_tablet_manager.cc:902] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:12.359056 14439 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:12.359966 14439 tablet_bootstrap.cc:492] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:12.360654 14439 ts_tablet_manager.cc:1367] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.073s user 0.021s sys 0.022s
I20240503 02:07:12.363463 14439 raft_consensus.cc:348] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.364437 14439 raft_consensus.cc:729] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:12.365142 14439 consensus_queue.cc:260] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.367570 14439 ts_tablet_manager.cc:1398] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240503 02:07:12.369414 13754 tablet_copy_service.cc:342] P 65b596345545434b9f09935907294192: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-ee01df95e61d4570a6c7827cd2acc525 received from {username='slave'} at 127.0.0.1:44274
I20240503 02:07:12.369859 13754 tablet_copy_service.cc:433] P 65b596345545434b9f09935907294192: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-ee01df95e61d4570a6c7827cd2acc525 on tablet ee01df95e61d4570a6c7827cd2acc525 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:12.455363 14375 raft_consensus.cc:1203] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:12.675141 14375 raft_consensus.cc:1203] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:12.812623 14438 tablet_copy_client.cc:308] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.3:35919
W20240503 02:07:12.853778 13709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.1:33671, user_credentials={real_user=slave}} blocked reactor thread for 92050.1us
W20240503 02:07:12.900424 13635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.3:35919, user_credentials={real_user=slave}} blocked reactor thread for 50080.3us
W20240503 02:07:12.924670 14444 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.262s sys 0.292s
W20240503 02:07:12.927649 14444 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.576s user 0.262s sys 0.292s
I20240503 02:07:12.937794 14444 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:12.938887 14444 tablet_bootstrap.cc:492] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:12.939604 14444 ts_tablet_manager.cc:1367] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.629s user 0.281s sys 0.297s
I20240503 02:07:12.942384 14444 raft_consensus.cc:348] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.943323 14444 raft_consensus.cc:729] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:12.944072 14444 consensus_queue.cc:260] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:12.953279 14444 ts_tablet_manager.cc:1398] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240503 02:07:12.960875 13754 tablet_copy_service.cc:342] P 65b596345545434b9f09935907294192: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-2fed43100e614122b8bbc0bc3506d664 received from {username='slave'} at 127.0.0.1:44274
I20240503 02:07:12.961303 13754 tablet_copy_service.cc:433] P 65b596345545434b9f09935907294192: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-2fed43100e614122b8bbc0bc3506d664 on tablet 2fed43100e614122b8bbc0bc3506d664 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:13.010342 14472 raft_consensus.cc:1050] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:13.012933 14472 consensus_queue.cc:237] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.078804 14375 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.080875 14474 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:13.106312 13811 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.110859 14473 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:13.111871 13828 tablet_copy_service.cc:140] P b518e4e142f54c789a56606b766c4e06: Received BeginTabletCopySession request for tablet b8461a9dc6344866b9e069d0b28ee64d from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:33690)
I20240503 02:07:13.112468 13828 tablet_copy_service.cc:161] P b518e4e142f54c789a56606b766c4e06: Beginning new tablet copy session on tablet b8461a9dc6344866b9e069d0b28ee64d from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:33690: session id = bd5512acd2bf412db3c2fbcffbf3b111-b8461a9dc6344866b9e069d0b28ee64d
I20240503 02:07:13.113593 13737 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.120061 14464 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:13.120540 13828 tablet_copy_source_session.cc:205] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:13.126063 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8461a9dc6344866b9e069d0b28ee64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:13.151422 14438 tablet_copy_client.cc:783] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:13.155766 14438 tablet_copy_client.cc:647] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:13.164597 14438 tablet_copy_client.cc:523] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:13.172590 14438 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:13.189504 14481 raft_consensus.cc:1050] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:13.194986 14472 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.199653 13811 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.208050 14375 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.198076 14481 consensus_queue.cc:237] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.218874 13567 catalog_manager.cc:5484] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:13.231395 13741 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.245965 14438 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:13.246852 14438 tablet_bootstrap.cc:492] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:13.247453 14438 ts_tablet_manager.cc:1367] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.075s user 0.027s sys 0.005s
I20240503 02:07:13.249984 14438 raft_consensus.cc:348] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:13.250814 14438 raft_consensus.cc:729] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:13.254057 14375 raft_consensus.cc:1203] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:13.263821 14438 consensus_queue.cc:260] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:13.266654 14438 ts_tablet_manager.cc:1398] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20240503 02:07:13.269745 13662 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.270931 13811 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.271193 14465 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:13.272096 13828 tablet_copy_service.cc:342] P b518e4e142f54c789a56606b766c4e06: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-b8461a9dc6344866b9e069d0b28ee64d received from {username='slave'} at 127.0.0.1:33690
I20240503 02:07:13.272512 13828 tablet_copy_service.cc:433] P b518e4e142f54c789a56606b766c4e06: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-b8461a9dc6344866b9e069d0b28ee64d on tablet b8461a9dc6344866b9e069d0b28ee64d with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:13.272816 14460 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:13.273634 14375 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.276307 14460 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:13.296723 14465 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.301488 13660 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.310915 14375 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.328225 13811 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.348484 13563 catalog_manager.cc:5484] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.361397 13660 consensus_queue.cc:237] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.367583 13737 consensus_queue.cc:237] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.372843 14375 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.373445 13662 raft_consensus.cc:1261] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.374583 14465 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:13.375541 14460 consensus_queue.cc:1035] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:13.382957 14409 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.394439 13811 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.396847 13551 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee01df95e61d4570a6c7827cd2acc525 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:13.394457 13660 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.400842 14375 raft_consensus.cc:1261] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.395080 14374 raft_consensus.cc:2932] T ee01df95e61d4570a6c7827cd2acc525 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.411717 14464 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:13.416829 14336 consensus_queue.cc:1035] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:13.419771 14474 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.422248 13812 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.446939 13566 catalog_manager.cc:5484] T ee01df95e61d4570a6c7827cd2acc525 P 65b596345545434b9f09935907294192 reported cstate change: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:13.449352 13553 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:13.452255 14375 raft_consensus.cc:2932] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.464066 13567 catalog_manager.cc:5484] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:13.466607 13807 tablet_service.cc:1514] Processing DeleteTablet for tablet ee01df95e61d4570a6c7827cd2acc525 with delete_type TABLET_DATA_TOMBSTONED (TS b518e4e142f54c789a56606b766c4e06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:07:13.482110 14374 raft_consensus.cc:1203] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:13.492821 13729 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 with delete_type TABLET_DATA_TOMBSTONED (TS 65b596345545434b9f09935907294192 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54470
I20240503 02:07:13.504391 14494 tablet_replica.cc:330] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: stopping tablet replica
I20240503 02:07:13.510506 14494 raft_consensus.cc:2229] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:13.510831 14493 tablet_replica.cc:330] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: stopping tablet replica
I20240503 02:07:13.511287 14494 raft_consensus.cc:2258] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:13.528663 14493 raft_consensus.cc:2229] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:13.529600 14493 raft_consensus.cc:2258] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:13.532685 14494 ts_tablet_manager.cc:1875] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:13.541141 14493 ts_tablet_manager.cc:1875] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:13.559379 14494 ts_tablet_manager.cc:1888] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:13.560040 14494 log.cc:1196] T 1ecb6a74b50e41e78a9e7d84d93b80d8 P 65b596345545434b9f09935907294192: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/wals/1ecb6a74b50e41e78a9e7d84d93b80d8
I20240503 02:07:13.562237 13551 catalog_manager.cc:4830] TS 65b596345545434b9f09935907294192 (127.12.53.2:42457): tablet 1ecb6a74b50e41e78a9e7d84d93b80d8 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:13.570010 14493 ts_tablet_manager.cc:1888] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:13.570636 14493 log.cc:1196] T ee01df95e61d4570a6c7827cd2acc525 P b518e4e142f54c789a56606b766c4e06: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/wals/ee01df95e61d4570a6c7827cd2acc525
I20240503 02:07:13.573462 13554 catalog_manager.cc:4830] TS b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): tablet ee01df95e61d4570a6c7827cd2acc525 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:13.844244 14465 raft_consensus.cc:1050] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:13.846381 14465 consensus_queue.cc:237] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.860565 13660 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.861018 13812 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.862242 14374 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.862639 14481 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:13.864030 14409 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:13.869985 14481 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:13.891242 14465 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.894012 14375 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.896816 14476 raft_consensus.cc:1050] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:13.899124 14476 consensus_queue.cc:237] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.893944 13811 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.912014 13567 catalog_manager.cc:5484] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.937922 13660 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:13.939035 14375 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.955818 14093 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:13.964627 13737 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:13.965327 13743 consensus_queue.cc:237] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:13.973716 14332 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240503 02:07:13.986081 14374 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.988103 14465 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:13.988766 13811 raft_consensus.cc:1261] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:13.991909 14465 consensus_queue.cc:1035] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:14.002408 14409 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.016587 14374 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.020110 13812 raft_consensus.cc:2932] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.034181 13660 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:14.035969 13551 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fed43100e614122b8bbc0bc3506d664 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:14.041152 14332 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:14.057193 14435 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
W20240503 02:07:14.067610 13551 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 2fed43100e614122b8bbc0bc3506d664 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 65b596345545434b9f09935907294192 (127.12.53.2:42457) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240503 02:07:14.066279 13739 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.089784 13567 catalog_manager.cc:5484] T 2fed43100e614122b8bbc0bc3506d664 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.096556 14374 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.101398 13663 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.120172 13658 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed43100e614122b8bbc0bc3506d664 with delete_type TABLET_DATA_TOMBSTONED (TS 7da64b3eda0a40b6b183c1cfc19db738 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54568
I20240503 02:07:14.119634 13565 catalog_manager.cc:5484] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.123684 14504 tablet_replica.cc:330] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: stopping tablet replica
I20240503 02:07:14.124915 14504 raft_consensus.cc:2229] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:14.125650 14504 raft_consensus.cc:2258] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:14.134011 14504 ts_tablet_manager.cc:1875] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:14.158179 13816 consensus_queue.cc:237] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:14.163517 14374 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:14.165199 13739 raft_consensus.cc:1261] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:14.165118 14093 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:14.166673 14475 consensus_queue.cc:1035] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:14.169256 14504 ts_tablet_manager.cc:1888] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:14.169798 14504 log.cc:1196] T 2fed43100e614122b8bbc0bc3506d664 P 7da64b3eda0a40b6b183c1cfc19db738: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/wals/2fed43100e614122b8bbc0bc3506d664
I20240503 02:07:14.171593 13553 catalog_manager.cc:4830] TS 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): tablet 2fed43100e614122b8bbc0bc3506d664 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:14.174021 14093 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.177870 14374 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.178978 13739 raft_consensus.cc:2932] T b8461a9dc6344866b9e069d0b28ee64d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:14.187846 13554 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8461a9dc6344866b9e069d0b28ee64d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:14.198058 13567 catalog_manager.cc:5484] T b8461a9dc6344866b9e069d0b28ee64d P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:14.207783 13658 tablet_service.cc:1514] Processing DeleteTablet for tablet b8461a9dc6344866b9e069d0b28ee64d with delete_type TABLET_DATA_TOMBSTONED (TS 7da64b3eda0a40b6b183c1cfc19db738 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54568
I20240503 02:07:14.208712 14504 tablet_replica.cc:330] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: stopping tablet replica
I20240503 02:07:14.210055 14504 raft_consensus.cc:2229] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:14.210650 14504 raft_consensus.cc:2258] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:14.216894 14504 ts_tablet_manager.cc:1875] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:14.228875 14504 ts_tablet_manager.cc:1888] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:14.229210 14504 log.cc:1196] T b8461a9dc6344866b9e069d0b28ee64d P 7da64b3eda0a40b6b183c1cfc19db738: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/wals/b8461a9dc6344866b9e069d0b28ee64d
I20240503 02:07:14.230338 13553 catalog_manager.cc:4830] TS 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): tablet b8461a9dc6344866b9e069d0b28ee64d (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:15.745352 13739 consensus_queue.cc:237] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:15.751358 13818 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.751366 13660 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:15.751541 13710 consensus_peers.cc:485] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07abdb29bc6d463c8928a23e8579ebac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:15.752681 14481 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.753556 14409 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.761459 14481 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.763001 13660 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.772361 13818 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.774893 13664 consensus_queue.cc:237] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:15.780655 13563 catalog_manager.cc:5484] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:07:15.791734 13635 consensus_peers.cc:485] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a71d633c024e481d9dcd7dc94259a65d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:15.792317 13818 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.792498 13739 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.793828 14464 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.794819 14509 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.804715 14464 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.806931 13739 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.809826 13818 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.821204 13816 consensus_queue.cc:237] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:15.823761 13567 catalog_manager.cc:5484] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.828850 13739 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.831611 14514 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:07:15.833810 13784 consensus_peers.cc:485] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a24625c57a084e26a113238421e92823. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:15.836858 13664 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.838480 14525 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.842527 14525 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.846048 13739 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.856452 13816 consensus_queue.cc:237] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:15.861214 13566 catalog_manager.cc:5484] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:15.865307 13664 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
W20240503 02:07:15.881343 13784 consensus_peers.cc:485] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4459e1c4e013400f920bba94d281d95e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:15.882112 13739 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.883591 13664 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:15.883543 14332 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.885385 14332 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:15.898276 14525 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.899802 13739 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.903101 13664 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:15.915726 13563 catalog_manager.cc:5484] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:16.139290 14542 ts_tablet_manager.cc:902] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 65b596345545434b9f09935907294192 (127.12.53.2:42457)
W20240503 02:07:16.139652 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080024fda0 after lost signal to thread 12503
W20240503 02:07:16.140395 13598 debug-util.cc:398] Leaking SignalData structure 0x7b0800234080 after lost signal to thread 13601
W20240503 02:07:16.141461 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080003ed00 after lost signal to thread 13693
W20240503 02:07:16.142452 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080024f800 after lost signal to thread 13768
W20240503 02:07:16.143491 13598 debug-util.cc:398] Leaking SignalData structure 0x7b08002a4a00 after lost signal to thread 13842
I20240503 02:07:16.144023 14542 tablet_copy_client.cc:308] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.2:42457
W20240503 02:07:16.145359 13598 debug-util.cc:398] Leaking SignalData structure 0x7b080030f300 after lost signal to thread 14399
W20240503 02:07:16.145800 13598 debug-util.cc:398] Leaking SignalData structure 0x7b08002eb7c0 after lost signal to thread 14495
I20240503 02:07:16.145920 13754 tablet_copy_service.cc:140] P 65b596345545434b9f09935907294192: Received BeginTabletCopySession request for tablet 07abdb29bc6d463c8928a23e8579ebac from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:44274)
W20240503 02:07:16.146142 13598 debug-util.cc:398] Leaking SignalData structure 0x7b08002eb820 after lost signal to thread 14498
I20240503 02:07:16.146526 13754 tablet_copy_service.cc:161] P 65b596345545434b9f09935907294192: Beginning new tablet copy session on tablet 07abdb29bc6d463c8928a23e8579ebac from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:44274: session id = bd5512acd2bf412db3c2fbcffbf3b111-07abdb29bc6d463c8928a23e8579ebac
I20240503 02:07:16.167093 13754 tablet_copy_source_session.cc:205] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:16.175634 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07abdb29bc6d463c8928a23e8579ebac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:16.196053 14542 tablet_copy_client.cc:783] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:16.200042 14542 tablet_copy_client.cc:647] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:16.204102 14542 tablet_copy_client.cc:523] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:16.217479 14542 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:16.246325 14542 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:16.247188 14542 tablet_bootstrap.cc:492] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:16.248014 14542 ts_tablet_manager.cc:1367] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.003s
I20240503 02:07:16.250682 14542 raft_consensus.cc:348] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.251636 14542 raft_consensus.cc:729] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:16.252316 14542 consensus_queue.cc:260] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.255373 14542 ts_tablet_manager.cc:1398] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 02:07:16.256963 13754 tablet_copy_service.cc:342] P 65b596345545434b9f09935907294192: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-07abdb29bc6d463c8928a23e8579ebac received from {username='slave'} at 127.0.0.1:44274
I20240503 02:07:16.257339 13754 tablet_copy_service.cc:433] P 65b596345545434b9f09935907294192: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-07abdb29bc6d463c8928a23e8579ebac on tablet 07abdb29bc6d463c8928a23e8579ebac with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:16.275348 14542 ts_tablet_manager.cc:902] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671)
I20240503 02:07:16.297649 14542 tablet_copy_client.cc:308] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.1:33671
I20240503 02:07:16.299997 13679 tablet_copy_service.cc:140] P 7da64b3eda0a40b6b183c1cfc19db738: Received BeginTabletCopySession request for tablet a71d633c024e481d9dcd7dc94259a65d from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:55890)
I20240503 02:07:16.300496 13679 tablet_copy_service.cc:161] P 7da64b3eda0a40b6b183c1cfc19db738: Beginning new tablet copy session on tablet a71d633c024e481d9dcd7dc94259a65d from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:55890: session id = bd5512acd2bf412db3c2fbcffbf3b111-a71d633c024e481d9dcd7dc94259a65d
I20240503 02:07:16.306160 13679 tablet_copy_source_session.cc:205] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:16.308622 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d633c024e481d9dcd7dc94259a65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:16.322999 14553 ts_tablet_manager.cc:902] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:16.322760 14542 tablet_copy_client.cc:783] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:16.323956 14542 tablet_copy_client.cc:647] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:16.327329 14542 tablet_copy_client.cc:523] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:16.335878 14542 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:16.347519 14553 tablet_copy_client.cc:308] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.3:35919
I20240503 02:07:16.349507 13828 tablet_copy_service.cc:140] P b518e4e142f54c789a56606b766c4e06: Received BeginTabletCopySession request for tablet a24625c57a084e26a113238421e92823 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:33690)
I20240503 02:07:16.350060 13828 tablet_copy_service.cc:161] P b518e4e142f54c789a56606b766c4e06: Beginning new tablet copy session on tablet a24625c57a084e26a113238421e92823 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:33690: session id = bd5512acd2bf412db3c2fbcffbf3b111-a24625c57a084e26a113238421e92823
I20240503 02:07:16.357030 13828 tablet_copy_source_session.cc:205] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:16.359970 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24625c57a084e26a113238421e92823. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:16.366346 14542 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:16.367364 14542 tablet_bootstrap.cc:492] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:16.368027 14542 ts_tablet_manager.cc:1367] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.005s
I20240503 02:07:16.370770 14542 raft_consensus.cc:348] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.371598 14542 raft_consensus.cc:729] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:16.372298 14542 consensus_queue.cc:260] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.376050 14542 ts_tablet_manager.cc:1398] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:07:16.377112 14553 tablet_copy_client.cc:783] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:16.377718 14553 tablet_copy_client.cc:647] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:16.377735 13679 tablet_copy_service.cc:342] P 7da64b3eda0a40b6b183c1cfc19db738: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a71d633c024e481d9dcd7dc94259a65d received from {username='slave'} at 127.0.0.1:55890
I20240503 02:07:16.378330 13679 tablet_copy_service.cc:433] P 7da64b3eda0a40b6b183c1cfc19db738: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a71d633c024e481d9dcd7dc94259a65d on tablet a71d633c024e481d9dcd7dc94259a65d with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:16.383098 14553 tablet_copy_client.cc:523] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:16.391714 14553 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:16.411613 14553 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:16.412415 14553 tablet_bootstrap.cc:492] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:16.413061 14553 ts_tablet_manager.cc:1367] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.004s
I20240503 02:07:16.415249 14553 raft_consensus.cc:348] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.415849 14553 raft_consensus.cc:729] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:16.416380 14553 consensus_queue.cc:260] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.418619 14553 ts_tablet_manager.cc:1398] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:07:16.420171 13828 tablet_copy_service.cc:342] P b518e4e142f54c789a56606b766c4e06: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a24625c57a084e26a113238421e92823 received from {username='slave'} at 127.0.0.1:33690
I20240503 02:07:16.420567 13828 tablet_copy_service.cc:433] P b518e4e142f54c789a56606b766c4e06: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a24625c57a084e26a113238421e92823 on tablet a24625c57a084e26a113238421e92823 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:16.501673 14553 ts_tablet_manager.cc:902] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:16.512264 14553 tablet_copy_client.cc:308] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.3:35919
I20240503 02:07:16.514154 13828 tablet_copy_service.cc:140] P b518e4e142f54c789a56606b766c4e06: Received BeginTabletCopySession request for tablet 4459e1c4e013400f920bba94d281d95e from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:33690)
I20240503 02:07:16.514726 13828 tablet_copy_service.cc:161] P b518e4e142f54c789a56606b766c4e06: Beginning new tablet copy session on tablet 4459e1c4e013400f920bba94d281d95e from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:33690: session id = bd5512acd2bf412db3c2fbcffbf3b111-4459e1c4e013400f920bba94d281d95e
I20240503 02:07:16.521862 13828 tablet_copy_source_session.cc:205] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:16.524495 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4459e1c4e013400f920bba94d281d95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:16.537207 14553 tablet_copy_client.cc:783] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:16.537791 14553 tablet_copy_client.cc:647] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:16.545339 14553 tablet_copy_client.cc:523] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:16.553300 14553 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:16.570335 14553 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:16.571089 14553 tablet_bootstrap.cc:492] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:16.571696 14553 ts_tablet_manager.cc:1367] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240503 02:07:16.574208 14553 raft_consensus.cc:348] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.574890 14553 raft_consensus.cc:729] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:16.575443 14553 consensus_queue.cc:260] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:16.577966 14553 ts_tablet_manager.cc:1398] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 02:07:16.579520 13828 tablet_copy_service.cc:342] P b518e4e142f54c789a56606b766c4e06: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-4459e1c4e013400f920bba94d281d95e received from {username='slave'} at 127.0.0.1:33690
I20240503 02:07:16.579910 13828 tablet_copy_service.cc:433] P b518e4e142f54c789a56606b766c4e06: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-4459e1c4e013400f920bba94d281d95e on tablet 4459e1c4e013400f920bba94d281d95e with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:16.655901 14374 raft_consensus.cc:1203] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:16.662750 14374 raft_consensus.cc:1203] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:16.877983 14374 raft_consensus.cc:1203] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:17.064690 14505 raft_consensus.cc:1050] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:17.067458 14505 consensus_queue.cc:237] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.075716 13660 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.075874 14374 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.077172 14565 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.078132 14522 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.078683 13818 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.080659 14565 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.088220 14522 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.094439 13816 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.102876 13660 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.105700 14374 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.111259 13567 catalog_manager.cc:5484] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:17.113816 14375 raft_consensus.cc:1203] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:17.142549 13739 consensus_queue.cc:237] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.146936 14374 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.147850 13660 raft_consensus.cc:1261] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.149449 14517 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.150552 14565 consensus_queue.cc:1035] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.156417 14522 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.160250 14374 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.160037 13660 raft_consensus.cc:2932] T 07abdb29bc6d463c8928a23e8579ebac P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.164083 14512 raft_consensus.cc:1050] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:17.166504 14512 consensus_queue.cc:237] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.178104 13551 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07abdb29bc6d463c8928a23e8579ebac with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:17.176211 13566 catalog_manager.cc:5484] T 07abdb29bc6d463c8928a23e8579ebac P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.181609 13816 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.182160 14374 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.183710 14512 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.184607 14509 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.188302 13741 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.194687 14566 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.213660 14509 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.216132 14374 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.224174 13816 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.230278 13807 tablet_service.cc:1514] Processing DeleteTablet for tablet 07abdb29bc6d463c8928a23e8579ebac with delete_type TABLET_DATA_TOMBSTONED (TS b518e4e142f54c789a56606b766c4e06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:07:17.255770 13743 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.235752 13566 catalog_manager.cc:5484] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:17.274247 14581 tablet_replica.cc:330] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: stopping tablet replica
I20240503 02:07:17.275369 14581 raft_consensus.cc:2229] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:17.276113 14581 raft_consensus.cc:2258] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:17.295694 13664 consensus_queue.cc:237] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.298200 14581 ts_tablet_manager.cc:1875] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:17.302291 14374 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.303755 14577 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.306982 13818 raft_consensus.cc:1261] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.308144 14577 consensus_queue.cc:1035] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.313371 14509 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.320060 14374 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.321811 13553 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a71d633c024e481d9dcd7dc94259a65d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:17.323302 13818 raft_consensus.cc:2932] T a71d633c024e481d9dcd7dc94259a65d P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.329403 13565 catalog_manager.cc:5484] T a71d633c024e481d9dcd7dc94259a65d P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:17.339671 14581 ts_tablet_manager.cc:1888] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:17.340451 13729 tablet_service.cc:1514] Processing DeleteTablet for tablet a71d633c024e481d9dcd7dc94259a65d with delete_type TABLET_DATA_TOMBSTONED (TS 65b596345545434b9f09935907294192 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54470
I20240503 02:07:17.349078 14581 log.cc:1196] T 07abdb29bc6d463c8928a23e8579ebac P b518e4e142f54c789a56606b766c4e06: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/wals/07abdb29bc6d463c8928a23e8579ebac
I20240503 02:07:17.353986 13554 catalog_manager.cc:4830] TS b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): tablet 07abdb29bc6d463c8928a23e8579ebac (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:17.357645 14582 tablet_replica.cc:330] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: stopping tablet replica
I20240503 02:07:17.358569 14582 raft_consensus.cc:2229] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:17.359139 14582 raft_consensus.cc:2258] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:17.362280 14582 ts_tablet_manager.cc:1875] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:17.376930 14582 ts_tablet_manager.cc:1888] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:17.377269 14582 log.cc:1196] T a71d633c024e481d9dcd7dc94259a65d P 65b596345545434b9f09935907294192: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/wals/a71d633c024e481d9dcd7dc94259a65d
I20240503 02:07:17.377629 14475 raft_consensus.cc:1050] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:17.379992 14475 consensus_queue.cc:237] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.381207 13551 catalog_manager.cc:4830] TS 65b596345545434b9f09935907294192 (127.12.53.2:42457): tablet a71d633c024e481d9dcd7dc94259a65d (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:17.385573 13743 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.386762 14514 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.387243 13664 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.388386 14475 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.391497 14374 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.392946 14567 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.412932 14514 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.414394 13664 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.423951 14375 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.431740 13567 catalog_manager.cc:5484] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.449273 13735 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.458209 13818 consensus_queue.cc:237] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.464393 14375 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.466049 13664 raft_consensus.cc:1261] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.467099 14587 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.469353 14587 consensus_queue.cc:1035] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.473870 14475 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.475495 14375 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.481693 13554 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a24625c57a084e26a113238421e92823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:17.486389 13664 raft_consensus.cc:2932] T a24625c57a084e26a113238421e92823 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.488168 13565 catalog_manager.cc:5484] T a24625c57a084e26a113238421e92823 P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.497716 13729 tablet_service.cc:1514] Processing DeleteTablet for tablet a24625c57a084e26a113238421e92823 with delete_type TABLET_DATA_TOMBSTONED (TS 65b596345545434b9f09935907294192 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54470
I20240503 02:07:17.498689 14582 tablet_replica.cc:330] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: stopping tablet replica
I20240503 02:07:17.499666 14582 raft_consensus.cc:2229] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:17.500646 14582 raft_consensus.cc:2258] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:17.504585 14582 ts_tablet_manager.cc:1875] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:17.517953 14582 ts_tablet_manager.cc:1888] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:17.518445 14582 log.cc:1196] T a24625c57a084e26a113238421e92823 P 65b596345545434b9f09935907294192: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/wals/a24625c57a084e26a113238421e92823
I20240503 02:07:17.520085 13551 catalog_manager.cc:4830] TS 65b596345545434b9f09935907294192 (127.12.53.2:42457): tablet a24625c57a084e26a113238421e92823 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:17.653957 14589 raft_consensus.cc:1050] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:17.655970 14589 consensus_queue.cc:237] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.660997 13743 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.661852 14375 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.662181 14567 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.663019 13668 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:17.663748 14589 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:17.665311 14567 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:17.674947 14475 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.680085 13668 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.684715 14375 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.693977 13743 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.699801 13565 catalog_manager.cc:5484] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.722245 13814 consensus_queue.cc:237] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:17.731871 13735 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.733227 14589 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.733649 14375 raft_consensus.cc:1261] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:17.734930 14475 consensus_queue.cc:1035] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:17.740705 13735 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.742764 14375 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.739629 14589 raft_consensus.cc:2932] T 4459e1c4e013400f920bba94d281d95e P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.759675 13567 catalog_manager.cc:5484] T 4459e1c4e013400f920bba94d281d95e P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:17.767971 13554 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4459e1c4e013400f920bba94d281d95e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:17.775336 13658 tablet_service.cc:1514] Processing DeleteTablet for tablet 4459e1c4e013400f920bba94d281d95e with delete_type TABLET_DATA_TOMBSTONED (TS 7da64b3eda0a40b6b183c1cfc19db738 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54568
I20240503 02:07:17.777091 14594 tablet_replica.cc:330] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: stopping tablet replica
I20240503 02:07:17.778075 14594 raft_consensus.cc:2229] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:17.778616 14594 raft_consensus.cc:2258] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:17.781179 14594 ts_tablet_manager.cc:1875] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:17.796922 14594 ts_tablet_manager.cc:1888] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:17.797363 14594 log.cc:1196] T 4459e1c4e013400f920bba94d281d95e P 7da64b3eda0a40b6b183c1cfc19db738: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/wals/4459e1c4e013400f920bba94d281d95e
I20240503 02:07:17.798890 13553 catalog_manager.cc:4830] TS 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): tablet 4459e1c4e013400f920bba94d281d95e (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:18.832944 13668 consensus_queue.cc:237] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:18.848991 13814 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:18.849025 13635 consensus_peers.cc:485] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8e8e9e6dec94158989853c4dbb2787b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:18.849498 13743 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:18.850258 14512 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:18.851256 14509 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:18.858392 14509 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.859503 13743 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.860319 13814 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.879204 13563 catalog_manager.cc:5484] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.889554 13818 consensus_queue.cc:237] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:18.909165 13735 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:18.910295 13784 consensus_peers.cc:485] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e893a056b25c462d82b4147b937c8c94. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:18.910669 13668 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:18.911563 14589 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:18.912510 14595 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:18.930140 14589 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.939292 13735 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.945794 13743 consensus_queue.cc:237] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:18.948428 13668 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.956091 13812 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:18.957751 13660 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:18.956398 13566 catalog_manager.cc:5484] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:18.959383 14505 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:07:18.957999 13710 consensus_peers.cc:485] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97598aefb41c4115a37385063f8d0eb7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:18.965685 14522 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:18.986747 14522 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.987560 13814 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:18.998975 13664 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.006945 13563 catalog_manager.cc:5484] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.013164 13814 consensus_queue.cc:237] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.021196 13743 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:19.021695 13784 consensus_peers.cc:485] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2365517fa2742fead22e9bceb13af2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:19.022244 13664 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:19.022300 14475 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:19.023531 14609 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:19.033102 14475 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.036357 13664 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.040211 13743 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.056162 13566 catalog_manager.cc:5484] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:19.366572 14619 ts_tablet_manager.cc:902] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671)
I20240503 02:07:19.372354 14619 tablet_copy_client.cc:308] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.1:33671
I20240503 02:07:19.374141 13679 tablet_copy_service.cc:140] P 7da64b3eda0a40b6b183c1cfc19db738: Received BeginTabletCopySession request for tablet a8e8e9e6dec94158989853c4dbb2787b from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:55890)
I20240503 02:07:19.374682 13679 tablet_copy_service.cc:161] P 7da64b3eda0a40b6b183c1cfc19db738: Beginning new tablet copy session on tablet a8e8e9e6dec94158989853c4dbb2787b from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:55890: session id = bd5512acd2bf412db3c2fbcffbf3b111-a8e8e9e6dec94158989853c4dbb2787b
I20240503 02:07:19.384049 13679 tablet_copy_source_session.cc:205] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:19.387015 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e8e9e6dec94158989853c4dbb2787b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:19.400795 14619 tablet_copy_client.cc:783] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:19.401332 14619 tablet_copy_client.cc:647] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:19.404379 14619 tablet_copy_client.cc:523] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:19.411998 14619 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:19.437448 14619 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:19.439673 14619 tablet_bootstrap.cc:492] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:19.440320 14619 ts_tablet_manager.cc:1367] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20240503 02:07:19.442392 14619 raft_consensus.cc:348] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.443291 14619 raft_consensus.cc:729] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:19.443825 14619 consensus_queue.cc:260] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.446547 14619 ts_tablet_manager.cc:1398] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 02:07:19.448617 13679 tablet_copy_service.cc:342] P 7da64b3eda0a40b6b183c1cfc19db738: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a8e8e9e6dec94158989853c4dbb2787b received from {username='slave'} at 127.0.0.1:55890
I20240503 02:07:19.449038 13679 tablet_copy_service.cc:433] P 7da64b3eda0a40b6b183c1cfc19db738: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-a8e8e9e6dec94158989853c4dbb2787b on tablet a8e8e9e6dec94158989853c4dbb2787b with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:19.497735 14619 ts_tablet_manager.cc:902] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:19.501332 14619 tablet_copy_client.cc:308] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.3:35919
I20240503 02:07:19.502979 13828 tablet_copy_service.cc:140] P b518e4e142f54c789a56606b766c4e06: Received BeginTabletCopySession request for tablet b2365517fa2742fead22e9bceb13af2d from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:33690)
I20240503 02:07:19.503468 13828 tablet_copy_service.cc:161] P b518e4e142f54c789a56606b766c4e06: Beginning new tablet copy session on tablet b2365517fa2742fead22e9bceb13af2d from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:33690: session id = bd5512acd2bf412db3c2fbcffbf3b111-b2365517fa2742fead22e9bceb13af2d
I20240503 02:07:19.504528 14630 ts_tablet_manager.cc:902] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
I20240503 02:07:19.509249 14630 tablet_copy_client.cc:308] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.3:35919
I20240503 02:07:19.511050 13827 tablet_copy_service.cc:140] P b518e4e142f54c789a56606b766c4e06: Received BeginTabletCopySession request for tablet e893a056b25c462d82b4147b937c8c94 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:33690)
I20240503 02:07:19.511483 13827 tablet_copy_service.cc:161] P b518e4e142f54c789a56606b766c4e06: Beginning new tablet copy session on tablet e893a056b25c462d82b4147b937c8c94 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:33690: session id = bd5512acd2bf412db3c2fbcffbf3b111-e893a056b25c462d82b4147b937c8c94
I20240503 02:07:19.514943 13828 tablet_copy_source_session.cc:205] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:19.517922 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2365517fa2742fead22e9bceb13af2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:19.518939 13827 tablet_copy_source_session.cc:205] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:19.522998 14630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893a056b25c462d82b4147b937c8c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:19.533095 14619 tablet_copy_client.cc:783] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:19.533658 14619 tablet_copy_client.cc:647] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:19.537647 14630 tablet_copy_client.cc:783] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:19.538249 14630 tablet_copy_client.cc:647] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:19.540160 14619 tablet_copy_client.cc:523] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:19.542749 14630 tablet_copy_client.cc:523] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:19.548418 14619 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:19.550081 14630 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:19.576105 14630 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:19.578121 14630 tablet_bootstrap.cc:492] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:19.578791 14630 ts_tablet_manager.cc:1367] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.012s
I20240503 02:07:19.582079 14638 ts_tablet_manager.cc:902] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 65b596345545434b9f09935907294192 (127.12.53.2:42457)
I20240503 02:07:19.581852 14630 raft_consensus.cc:348] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.582737 14630 raft_consensus.cc:729] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:19.583403 14630 consensus_queue.cc:260] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.587668 14638 tablet_copy_client.cc:308] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Beginning tablet copy session from remote peer at address 127.12.53.2:42457
I20240503 02:07:19.589068 13754 tablet_copy_service.cc:140] P 65b596345545434b9f09935907294192: Received BeginTabletCopySession request for tablet 97598aefb41c4115a37385063f8d0eb7 from peer bd5512acd2bf412db3c2fbcffbf3b111 ({username='slave'} at 127.0.0.1:44274)
I20240503 02:07:19.589530 14630 ts_tablet_manager.cc:1398] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:07:19.589604 13754 tablet_copy_service.cc:161] P 65b596345545434b9f09935907294192: Beginning new tablet copy session on tablet 97598aefb41c4115a37385063f8d0eb7 from peer bd5512acd2bf412db3c2fbcffbf3b111 at {username='slave'} at 127.0.0.1:44274: session id = bd5512acd2bf412db3c2fbcffbf3b111-97598aefb41c4115a37385063f8d0eb7
I20240503 02:07:19.592133 13827 tablet_copy_service.cc:342] P b518e4e142f54c789a56606b766c4e06: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-e893a056b25c462d82b4147b937c8c94 received from {username='slave'} at 127.0.0.1:33690
I20240503 02:07:19.592530 13827 tablet_copy_service.cc:433] P b518e4e142f54c789a56606b766c4e06: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-e893a056b25c462d82b4147b937c8c94 on tablet e893a056b25c462d82b4147b937c8c94 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:19.593156 14619 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:19.594031 14619 tablet_bootstrap.cc:492] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:19.594695 14619 ts_tablet_manager.cc:1367] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.018s
I20240503 02:07:19.597085 13754 tablet_copy_source_session.cc:205] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:07:19.597453 14619 raft_consensus.cc:348] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.598341 14619 raft_consensus.cc:729] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:19.598984 14619 consensus_queue.cc:260] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.600061 14638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97598aefb41c4115a37385063f8d0eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:07:19.601397 14619 ts_tablet_manager.cc:1398] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 02:07:19.606482 13827 tablet_copy_service.cc:342] P b518e4e142f54c789a56606b766c4e06: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-b2365517fa2742fead22e9bceb13af2d received from {username='slave'} at 127.0.0.1:33690
I20240503 02:07:19.606837 13827 tablet_copy_service.cc:433] P b518e4e142f54c789a56606b766c4e06: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-b2365517fa2742fead22e9bceb13af2d on tablet b2365517fa2742fead22e9bceb13af2d with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:19.616436 14638 tablet_copy_client.cc:783] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 0 data blocks...
I20240503 02:07:19.617074 14638 tablet_copy_client.cc:647] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Starting download of 1 WAL segments...
I20240503 02:07:19.620724 14638 tablet_copy_client.cc:523] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:07:19.628113 14638 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap starting.
I20240503 02:07:19.644968 14638 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:07:19.645823 14638 tablet_bootstrap.cc:492] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Bootstrap complete.
I20240503 02:07:19.646528 14638 ts_tablet_manager.cc:1367] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.003s
I20240503 02:07:19.648612 14638 raft_consensus.cc:348] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.649251 14638 raft_consensus.cc:729] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd5512acd2bf412db3c2fbcffbf3b111, State: Initialized, Role: LEARNER
I20240503 02:07:19.649740 14638 consensus_queue.cc:260] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:19.651476 14638 ts_tablet_manager.cc:1398] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:07:19.654510 13754 tablet_copy_service.cc:342] P 65b596345545434b9f09935907294192: Request end of tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-97598aefb41c4115a37385063f8d0eb7 received from {username='slave'} at 127.0.0.1:44274
I20240503 02:07:19.654960 13754 tablet_copy_service.cc:433] P 65b596345545434b9f09935907294192: ending tablet copy session bd5512acd2bf412db3c2fbcffbf3b111-97598aefb41c4115a37385063f8d0eb7 on tablet 97598aefb41c4115a37385063f8d0eb7 with peer bd5512acd2bf412db3c2fbcffbf3b111
I20240503 02:07:19.769870 14375 raft_consensus.cc:1203] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:19.966585 14375 raft_consensus.cc:1203] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:20.003764 14375 raft_consensus.cc:1203] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:20.096515 14375 raft_consensus.cc:1203] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:07:20.158932 14644 raft_consensus.cc:1050] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:20.173836 14644 consensus_queue.cc:237] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.196756 14373 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.200317 13739 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.200317 13818 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.205394 14600 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240503 02:07:20.220446 14599 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.267339 14647 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
W20240503 02:07:20.277091 13781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.53.2:42457, user_credentials={real_user=slave}} blocked reactor thread for 55669.7us
I20240503 02:07:20.295073 13814 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.293593 14647 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.331247 14375 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.333582 13563 catalog_manager.cc:5484] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.349963 13735 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.386554 14522 raft_consensus.cc:1050] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:20.389153 14522 consensus_queue.cc:237] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.402070 14375 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.402884 13664 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.404368 14636 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.405033 13814 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.406577 14611 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.411686 14660 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.414768 13664 consensus_queue.cc:237] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.421447 14375 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.423663 13739 raft_consensus.cc:1261] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.427878 14600 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.429781 14600 consensus_queue.cc:1035] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.435616 14660 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.438299 13814 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.439543 14644 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.440425 13739 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.448575 14373 raft_consensus.cc:2932] T a8e8e9e6dec94158989853c4dbb2787b P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.450163 14375 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.460268 13563 catalog_manager.cc:5484] T a8e8e9e6dec94158989853c4dbb2787b P 65b596345545434b9f09935907294192 reported cstate change: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.463702 13669 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.471555 13553 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8e8e9e6dec94158989853c4dbb2787b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:20.478168 13807 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e8e9e6dec94158989853c4dbb2787b with delete_type TABLET_DATA_TOMBSTONED (TS b518e4e142f54c789a56606b766c4e06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:07:20.498416 14664 tablet_replica.cc:330] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: stopping tablet replica
I20240503 02:07:20.499645 14664 raft_consensus.cc:2229] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:20.500403 14664 raft_consensus.cc:2258] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:20.504730 14650 raft_consensus.cc:1050] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:20.505409 14664 ts_tablet_manager.cc:1875] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:20.509876 14650 consensus_queue.cc:237] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.514117 13563 catalog_manager.cc:5484] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:20.524385 13739 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.526158 14651 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.529292 14664 ts_tablet_manager.cc:1888] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:20.529920 14664 log.cc:1196] T a8e8e9e6dec94158989853c4dbb2787b P b518e4e142f54c789a56606b766c4e06: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/wals/a8e8e9e6dec94158989853c4dbb2787b
I20240503 02:07:20.530094 14375 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.535287 14609 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.535604 13669 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.541384 13735 consensus_queue.cc:237] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.541889 14645 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.551697 13554 catalog_manager.cc:4830] TS b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): tablet a8e8e9e6dec94158989853c4dbb2787b (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:20.561694 14373 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.565814 14660 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.576619 13669 raft_consensus.cc:1261] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer 65b596345545434b9f09935907294192: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.582188 14475 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.585515 14611 consensus_queue.cc:1035] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.587759 14636 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 65b596345545434b9f09935907294192 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.591694 14650 raft_consensus.cc:1050] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06: attempting to promote NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 to VOTER
I20240503 02:07:20.597690 13669 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.598769 14650 consensus_queue.cc:237] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.604354 13664 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.604422 14374 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.609871 14373 raft_consensus.cc:2932] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.659296 13551 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97598aefb41c4115a37385063f8d0eb7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:20.654361 13566 catalog_manager.cc:5484] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:20.672344 13818 consensus_queue.cc:237] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.684262 13564 catalog_manager.cc:5484] T 97598aefb41c4115a37385063f8d0eb7 P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 3 to 4, VOTER b518e4e142f54c789a56606b766c4e06 (127.12.53.3) evicted. New cstate: current_term: 1 leader_uuid: "65b596345545434b9f09935907294192" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.692302 14374 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.694424 13669 raft_consensus.cc:1261] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.694886 14609 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.697746 13807 tablet_service.cc:1514] Processing DeleteTablet for tablet 97598aefb41c4115a37385063f8d0eb7 with delete_type TABLET_DATA_TOMBSTONED (TS b518e4e142f54c789a56606b766c4e06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:07:20.698686 14664 tablet_replica.cc:330] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: stopping tablet replica
I20240503 02:07:20.699790 14664 raft_consensus.cc:2229] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:20.700528 14664 raft_consensus.cc:2258] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:20.703958 14664 ts_tablet_manager.cc:1875] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:20.696463 14609 consensus_queue.cc:1035] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240503 02:07:20.708449 13669 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.711344 14651 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.712947 14609 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:20.716876 13741 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.721709 14374 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 LEARNER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:07:20.731833 14645 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:07:20.738231 13664 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.740309 14609 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:07:20.750491 14664 ts_tablet_manager.cc:1888] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:20.751021 14664 log.cc:1196] T 97598aefb41c4115a37385063f8d0eb7 P b518e4e142f54c789a56606b766c4e06: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-2-root/wals/97598aefb41c4115a37385063f8d0eb7
I20240503 02:07:20.752177 13554 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2365517fa2742fead22e9bceb13af2d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:20.753044 13554 catalog_manager.cc:4830] TS b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919): tablet 97598aefb41c4115a37385063f8d0eb7 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:20.762715 14374 raft_consensus.cc:2932] T b2365517fa2742fead22e9bceb13af2d P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.770459 13564 catalog_manager.cc:5484] T b2365517fa2742fead22e9bceb13af2d P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index 3 to 4, VOTER 65b596345545434b9f09935907294192 (127.12.53.2) evicted. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:07:20.777962 14645 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.781292 13741 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.792279 13664 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.795898 13729 tablet_service.cc:1514] Processing DeleteTablet for tablet b2365517fa2742fead22e9bceb13af2d with delete_type TABLET_DATA_TOMBSTONED (TS 65b596345545434b9f09935907294192 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54470
I20240503 02:07:20.792279 14375 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.809967 13566 catalog_manager.cc:5484] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 reported cstate change: config changed from index 2 to 3, bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.815886 14678 tablet_replica.cc:330] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: stopping tablet replica
I20240503 02:07:20.818027 14678 raft_consensus.cc:2229] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:20.818776 14678 pending_rounds.cc:62] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Trying to abort 1 pending ops.
I20240503 02:07:20.819353 14678 pending_rounds.cc:68] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023419557922299904 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b2365517fa2742fead22e9bceb13af2d" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } } }
I20240503 02:07:20.823333 14678 raft_consensus.cc:2894] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:07:20.824720 14678 raft_consensus.cc:2866] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:07:20.826958 14678 raft_consensus.cc:2258] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:20.834654 13814 consensus_queue.cc:237] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } }
I20240503 02:07:20.835522 14678 ts_tablet_manager.cc:1875] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:20.842329 14375 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.843613 13739 raft_consensus.cc:1261] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:07:20.845077 14675 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 02:07:20.845947 14645 consensus_queue.cc:1035] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:07:20.851747 14678 ts_tablet_manager.cc:1888] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:20.852295 14678 log.cc:1196] T b2365517fa2742fead22e9bceb13af2d P 65b596345545434b9f09935907294192: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-1-root/wals/b2365517fa2742fead22e9bceb13af2d
I20240503 02:07:20.852561 14675 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.853945 13551 catalog_manager.cc:4830] TS 65b596345545434b9f09935907294192 (127.12.53.2:42457): tablet b2365517fa2742fead22e9bceb13af2d (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:20.854271 13739 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.856967 14375 raft_consensus.cc:2932] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.862763 13554 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e893a056b25c462d82b4147b937c8c94 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:07:20.871803 13565 catalog_manager.cc:5484] T e893a056b25c462d82b4147b937c8c94 P bd5512acd2bf412db3c2fbcffbf3b111 reported cstate change: config changed from index 3 to 4, VOTER 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1) evicted. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: false } } }
I20240503 02:07:20.880931 13658 tablet_service.cc:1514] Processing DeleteTablet for tablet e893a056b25c462d82b4147b937c8c94 with delete_type TABLET_DATA_TOMBSTONED (TS 7da64b3eda0a40b6b183c1cfc19db738 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54568
I20240503 02:07:20.888111 14679 tablet_replica.cc:330] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: stopping tablet replica
I20240503 02:07:20.889237 14679 raft_consensus.cc:2229] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:07:20.889997 14679 raft_consensus.cc:2258] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:07:20.893668 14679 ts_tablet_manager.cc:1875] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:07:20.906047 14679 ts_tablet_manager.cc:1888] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:07:20.906355 14679 log.cc:1196] T e893a056b25c462d82b4147b937c8c94 P 7da64b3eda0a40b6b183c1cfc19db738: Deleting WAL directory at /tmp/dist-test-taskLu0Isa/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714701959428988-12500-0/minicluster-data/ts-0-root/wals/e893a056b25c462d82b4147b937c8c94
I20240503 02:07:20.907780 13553 catalog_manager.cc:4830] TS 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671): tablet e893a056b25c462d82b4147b937c8c94 (table test-workload [id=817018b9773e40f5b40c601196a15177]) successfully deleted
I20240503 02:07:21.934233 13818 consensus_queue.cc:237] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
W20240503 02:07:21.940186 13784 consensus_peers.cc:485] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c2a2e3ecd3f407ca6cd7a32ac82e89b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:21.941196 13669 raft_consensus.cc:1261] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:21.941761 13739 raft_consensus.cc:1261] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:21.942770 14682 consensus_queue.cc:1035] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:21.943545 14675 consensus_queue.cc:1035] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:21.952797 14682 raft_consensus.cc:2932] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:21.954299 13669 raft_consensus.cc:2932] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:21.955024 13739 raft_consensus.cc:2932] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:21.967442 13669 consensus_queue.cc:237] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:21.972337 13564 catalog_manager.cc:5484] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } attrs { replace: true } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
W20240503 02:07:22.000067 13635 consensus_peers.cc:485] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b664b6227e0f4268a89935f722d56c8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:22.002336 13818 raft_consensus.cc:1261] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:22.003048 13739 raft_consensus.cc:1261] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer 7da64b3eda0a40b6b183c1cfc19db738: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:22.004637 14600 consensus_queue.cc:1035] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240503 02:07:22.006826 14644 consensus_queue.cc:1035] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:22.021991 14600 raft_consensus.cc:2932] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.024992 13739 raft_consensus.cc:2932] T b664b6227e0f4268a89935f722d56c8f P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.029320 13818 raft_consensus.cc:2932] T b664b6227e0f4268a89935f722d56c8f P b518e4e142f54c789a56606b766c4e06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.038035 13814 consensus_queue.cc:237] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:22.041071 13565 catalog_manager.cc:5484] T b664b6227e0f4268a89935f722d56c8f P 7da64b3eda0a40b6b183c1cfc19db738 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "7da64b3eda0a40b6b183c1cfc19db738" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:22.065395 13741 raft_consensus.cc:1261] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:22.066835 14683 consensus_queue.cc:1035] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:22.067195 13669 raft_consensus.cc:1261] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:07:22.067751 13784 consensus_peers.cc:485] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f9ec7430d7f43baadcc6dd1296d7b28. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:22.068574 14682 consensus_queue.cc:1035] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:22.074383 14675 raft_consensus.cc:2932] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.076484 13741 raft_consensus.cc:2932] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.077250 13669 raft_consensus.cc:2932] T 4f9ec7430d7f43baadcc6dd1296d7b28 P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.089681 13814 consensus_queue.cc:237] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } }
I20240503 02:07:22.091357 13563 catalog_manager.cc:5484] T 4f9ec7430d7f43baadcc6dd1296d7b28 P b518e4e142f54c789a56606b766c4e06 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:07:22.098237 13741 raft_consensus.cc:1261] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:22.101430 14609 consensus_queue.cc:1035] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:07:22.103771 13784 consensus_peers.cc:485] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 -> Peer bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4:41693): Couldn't send request to peer bd5512acd2bf412db3c2fbcffbf3b111. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2524d54d0a2b4efc84ca69b6057c97ef. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:07:22.112208 13669 raft_consensus.cc:1261] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Refusing update from remote peer b518e4e142f54c789a56606b766c4e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:07:22.114238 14609 consensus_queue.cc:1035] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:07:22.115617 14682 raft_consensus.cc:2932] T 2524d54d0a2b4efc84ca69b6057c97ef P b518e4e142f54c789a56606b766c4e06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.116901 13739 raft_consensus.cc:2932] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.128556 13669 raft_consensus.cc:2932] T 2524d54d0a2b4efc84ca69b6057c97ef P 7da64b3eda0a40b6b183c1cfc19db738 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.136227 13563 catalog_manager.cc:5484] T 2524d54d0a2b4efc84ca69b6057c97ef P 65b596345545434b9f09935907294192 reported cstate change: config changed from index -1 to 2, NON_VOTER bd5512acd2bf412db3c2fbcffbf3b111 (127.12.53.4) added. New cstate: current_term: 1 leader_uuid: "b518e4e142f54c789a56606b766c4e06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7da64b3eda0a40b6b183c1cfc19db738" member_type: VOTER last_known_addr { host: "127.12.53.1" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "65b596345545434b9f09935907294192" member_type: VOTER last_known_addr { host: "127.12.53.2" port: 42457 } } peers { permanent_uuid: "b518e4e142f54c789a56606b766c4e06" member_type: VOTER last_known_addr { host: "127.12.53.3" port: 35919 } } peers { permanent_uuid: "bd5512acd2bf412db3c2fbcffbf3b111" member_type: NON_VOTER last_known_addr { host: "127.12.53.4" port: 41693 } attrs { promote: true } } }
I20240503 02:07:22.466055 14701 ts_tablet_manager.cc:902] T 7c2a2e3ecd3f407ca6cd7a32ac82e89b P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
F20240503 02:07:22.467087 14701 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.queue_time_us, tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.queue_time_us) (tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.run_cpu_time_us, tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.run_cpu_time_us) (tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.run_wall_time_us, tablet-download-pool-07abdb29bc6d463c8928a23e8579ebac.run_wall_time_us) (tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.queue_time_us, tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.queue_time_us) (tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.run_cpu_time_us, tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.run_cpu_time_us) (tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.run_wall_time_us, tablet-download-pool-1ecb6a74b50e41e78a9e7d84d93b80d8.run_wall_time_us) (tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.queue_time_us, tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.queue_time_us) (tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.run_cpu_time_us, tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.run_cpu_time_us) (tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.run_wall_time_us, tablet-download-pool-2fed43100e614122b8bbc0bc3506d664.run_wall_time_us) (tablet-download-pool-4459e1c4e013400f920bba94d281d95e.queue_time_us, tablet-download-pool-4459e1c4e013400f920bba94d281d95e.queue_time_us) (tablet-download-pool-4459e1c4e013400f920bba94d281d95e.run_cpu_time_us, tablet-download-pool-4459e1c4e013400f920bba94d281d95e.run_cpu_time_us) (tablet-download-pool-4459e1c4e013400f920bba94d281d95e.run_wall_time_us, tablet-download-pool-4459e1c4e013400f920bba94d281d95e.run_wall_time_us) (tablet-download-pool-7c2a2e3ecd3f407ca6cd7a32ac82e89b.queue_time_us, tablet-download-pool-7c2a2e3ecd3f407ca6cd7a32ac82e89b.queue_time_us) (tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.queue_time_us, tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.queue_time_us) (tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.run_cpu_time_us, tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.run_cpu_time_us) (tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.run_wall_time_us, tablet-download-pool-97598aefb41c4115a37385063f8d0eb7.run_wall_time_us) (tablet-download-pool-a24625c57a084e26a113238421e92823.queue_time_us, tablet-download-pool-a24625c57a084e26a113238421e92823.queue_time_us) (tablet-download-pool-a24625c57a084e26a113238421e92823.run_cpu_time_us, tablet-download-pool-a24625c57a084e26a113238421e92823.run_cpu_time_us) (tablet-download-pool-a24625c57a084e26a113238421e92823.run_wall_time_us, tablet-download-pool-a24625c57a084e26a113238421e92823.run_wall_time_us) (tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.queue_time_us, tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.queue_time_us) (tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.run_cpu_time_us, tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.run_cpu_time_us) (tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.run_wall_time_us, tablet-download-pool-a71d633c024e481d9dcd7dc94259a65d.run_wall_time_us) (tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.queue_time_us, tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.queue_time_us) (tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.run_cpu_time_us, tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.run_cpu_time_us) (tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.run_wall_time_us, tablet-download-pool-a8e8e9e6dec94158989853c4dbb2787b.run_wall_time_us) (tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.queue_time_us, tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.queue_time_us) (tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.run_cpu_time_us, tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.run_cpu_time_us) (tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.run_wall_time_us, tablet-download-pool-b2365517fa2742fead22e9bceb13af2d.run_wall_time_us) (tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.queue_time_us, tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.queue_time_us) (tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.run_cpu_time_us, tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.run_cpu_time_us) (tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.run_wall_time_us, tablet-download-pool-b8461a9dc6344866b9e069d0b28ee64d.run_wall_time_us) (tablet-download-pool-e893a056b25c462d82b4147b937c8c94.queue_time_us, tablet-download-pool-e893a056b25c462d82b4147b937c8c94.queue_time_us) (tablet-download-pool-e893a056b25c462d82b4147b937c8c94.run_cpu_time_us, tablet-download-pool-e893a056b25c462d82b4147b937c8c94.run_cpu_time_us) (tablet-download-pool-e893a056b25c462d82b4147b937c8c94.run_wall_time_us, tablet-download-pool-e893a056b25c462d82b4147b937c8c94.run_wall_time_us) (tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.queue_time_us, tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.queue_time_us) (tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.run_cpu_time_us, tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.run_cpu_time_us) (tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.run_wall_time_us, tablet-download-pool-ee01df95e61d4570a6c7827cd2acc525.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714702042 (unix time) try "date -d @1714702042" if you are using GNU date ***
I20240503 02:07:22.491359 14702 ts_tablet_manager.cc:902] T 4f9ec7430d7f43baadcc6dd1296d7b28 P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000030d4) received by PID 12500 (TID 0x7f6f52a46700) from PID 12500; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240503 02:07:22.509721 14703 ts_tablet_manager.cc:902] T b664b6227e0f4268a89935f722d56c8f P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer 7da64b3eda0a40b6b183c1cfc19db738 (127.12.53.1:33671)
@ 0x7f702159f980 (unknown) at ??:0
@ 0x7f701ab13fb7 gsignal at ??:0
@ 0x7f701ab15921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f701d909da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f701d90a950 google::LogMessage::Flush() at ??:0
@ 0x7f701d90fa8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f701e2973c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f701e26d94f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f701e26b0a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:07:22.711194 14716 ts_tablet_manager.cc:902] T 2524d54d0a2b4efc84ca69b6057c97ef P bd5512acd2bf412db3c2fbcffbf3b111: Initiating tablet copy from peer b518e4e142f54c789a56606b766c4e06 (127.12.53.3:35919)
@ 0x7f702608b422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7026091c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f70261274ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f70261583cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f702613e1ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f702613e17a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7f702613e10a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f702613e0d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f702613d2fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f70274e3da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f70274e3bda std::__1::function<>::operator()() at ??:0
@ 0x7f701e26f237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f701e272692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f701e27264a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f701e2725da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f701e2725a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f701e27189e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f70274e3da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f70274e3bda std::__1::function<>::operator()() at ??:0
@ 0x7f701e251937 kudu::Thread::SuperviseThread() at ??:0